Monitor pl-en training

[kpu]

Running in screen 86852 on the second half of alvis. It's currently in student training.

[ugermann]

Seems to have finished, but I'm a little concerned about the backtranslations that apparently were used to train the teachers.

ermann@alvis:~$ zcat /fs/surtr0/nbogoych/data/data/pl-en/pl-en-prod/translated/mono.pl.gz | head 
Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale
Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale Ale

[kpu]

Oh my that is terrible. The whole file is Ale with various lengths. @eu9ene the pipeline shouldn't be continuing if quality is that terrible.

[XapaJIaMnu]

I have seen such an issue before, it's fp16 mode being broken for some reason. Disable fp16 mode in the configuration file for plen and, sigh, delete the translated files.... I may try to do this tmrw.

[ugermann]

I copied the whole stuff to /fs/surtr/germann/bergamot/train/plen and am currently re-running things from backtranslation onwards on alvis.

[evgenyrp]

I also discovered this issue while retraining my models. It was fixed a month ago https://github.com/mozilla/firefox-translations-training/blob/22a3751a09dfdb2ba52f4d08c285e424c533dcde/configs/config.prod.yml#L64

I created an issue mozilla/translations#78 to stop training if the quality is too low.

[evgenyrp]

There is also another issue mozilla/translations#75. I don't know how your training is configured, but when I was retraining ru,pt,it -> en models, the quality of neither fine-tuned teacher was improved on top of the pretrained teacher, and even worse, it degraded and those models were used further in the pipeline. I tried to increase patience but it didn't help. We should investigate this.