Apple researchers find bilingual pre-training beats hyperparameter tuning for low-resource languages

For most languages of the world, language model pre-training operates in a data-constrained regime where models must repeat their training data many times, degrading generalization.