Possible overfitting with "TransCoder_model_1.pth" from Java -> Python

[GiovaniGuizzo]

I have been using TransCoder for a few research papers lately and I came across an interesting phenomenon using the pretrained model "TransCoder_model_1.pth" you provide in your docs.

For the program in your dataset COUNT_OF_PAIRS_SATISFYING_THE_GIVEN_CONDITION.java when translated to Python, it passes all test cases. We checked and the translation seems correct. However, when we perform any small syntactical change to the original Java code and try to translate again, the translation mismatches the changed program.

For example, we introduced a change to the following Java code by modifying s.charAt(i) -> s.charAt(++i) at line 5:

static  int f_gold( int a, int b ) {
    java.lang.String s = String.valueOf( b );
    int i;
    for (i = 0; i < s.length(); i++) {
        if (s.charAt( ++i ) != '9') {
            break;
        }
    }
    int result;
    if (i == s.length()) {
        result = a * s.length();
    } else {
        result = a * (s.length() - 1);
    }
    return result;
}

The translated code is as follows:

def f_gold(cls, a, b):
    """ generated source for method f_gold """
    s = String.valueOf(b)
    i = int()
    while i < len(s):
        i += 1
        if s.charAt(i) != '9':
            break
        i += 1
    result = int()
    if i == len(s):
        result = a * len(s)
    else:
        result = a * (1 - len(s))
    return result

Notice that the changed code is rightfully translated, however, result = a * (s.length() - 1); is now wrongfully translated to result = a * (1 - len(s)).

We have generated 89 variations of this Java function using MuJava, and this translation bug appears in every single one of them, except the original program.

Maybe this is an overfitting example. What are your thoughts?