Source code doesnt provide operation of data prepare for BERT-like models

[pikaliov]

Repository doesnt contain code for ALBERT/BERT: masking input tokens, masked labels, madked positions.
Did you follow hugginface or tensorflow.official algorithm to create dataset for train/eval?

[jarednielsen]

I'd recommend following the instructions provided by Nvidia: https://github.com/NVIDIA/DeepLearningExamples/tree/b7903f0f62b1cdc3356d27956b5c8dee3896f68d/TensorFlow/LanguageModeling/BERT#getting-the-data. Those TFRecords are the expected format for the training scripts.