Massively Multilingual Adaptation of Large Language Models Using Bilingual Translation Data

TL;DR

This study explores the impact of bilingual translation data on the performance of large multilingual language models, demonstrating improvements especially for low-resource languages through extensive pre-training and evaluation.

Contribution

It introduces the MaLA bilingual translation corpus and the EMMA-500 Llama 3 models, showing how bilingual data enhances multilingual model performance.

Findings

Bilingual data improves language transfer and performance.

Low-resource languages benefit significantly from bilingual pre-training.

Open-source resources facilitate further research.

Abstract

This paper investigates a critical design decision in the practice of massively multilingual continual pre-training -- the inclusion of parallel data. Specifically, we study the impact of bilingual translation data for massively multilingual language adaptation of the Llama3 family of models to 500 languages. To this end, we construct the MaLA bilingual translation corpus, containing data from more than 2,500 language pairs. Subsequently, we develop the EMMA-500 Llama 3 suite of four massively multilingual models -- continually pre-trained from the Llama 3 family of base models extensively on diverse data mixes up to 671B tokens -- and explore the effect of continual pre-training with or without bilingual translation data. Comprehensive evaluation across 7 tasks and 12 benchmarks demonstrates that bilingual data tends to enhance language transfer and performance, particularly for low-resource languages. We open-source the MaLA corpus, EMMA-500 Llama 3 suite artefacts, code, and model generations.