Large language models have provided convincing evidence of their significant capabilities, both in downstream tasks and in real-world scenarios. However, low- and mid-resource languages lack the necessary resources to train such models from scratch and often have to rely on multilingual models despite being underrepresented in the training data.
This talk describes the experiments that prove that, for mid-resource languages, continued pre-training with vocabulary adaptation is a better alternative resulting in a significant improvement over a random initialization of weights. Irene and Joan perform a comprehensive evaluation to assess the effectiveness of the different techniques to perform continued pre-training from an existing model. Vocabulary adaptation proves to be the most effective technique in terms of performance gains and training efficiency.
Add comment