Multilingual Pretraining Using a Large Corpus Machine-Translated from a Single Source Language

TL;DR

This paper demonstrates that machine-translated web data from a single high-quality source language can effectively pretrain multilingual LLMs, achieving competitive performance with significantly less data than existing models.

Contribution

It introduces a new multilingual pretraining dataset created via machine translation from English and shows that models trained on it perform on par or better than larger models trained on more data.

Findings

CuatroLLM matches or outperforms state-of-the-art multilingual models.

Using less than 6% of the data, CuatroLLM achieves comparable results.

Additional domain-specific pretraining improves multilingual reasoning performance.

Abstract

English, as a very high-resource language, enables the pretraining of high-quality large language models (LLMs). The same cannot be said for most other languages, as leading LLMs still underperform for non-English languages, likely due to a gap in the quality and diversity of the available multilingual pretraining corpora. In this work, we find that machine-translated text from a single high-quality source language can contribute significantly to the pretraining of multilingual LLMs. We translate FineWeb-Edu, a high-quality English web dataset, into French, German, and Spanish, resulting in a final 300B-token dataset, which we call TransWeb-Edu, and pretrain a 1.3B-parameter model, CuatroLLM, from scratch on this dataset. Across five non-English reasoning tasks, we show that CuatroLLM matches or outperforms state-of-the-art multilingual models trained using closed data, such as Llama3.2 and Gemma2, despite using an order of magnitude less data, such as about 6% of the tokens used for Llama3.2's training. We further demonstrate that with additional domain-specific pretraining, amounting to less than 1% of TransWeb-Edu, CuatroLLM surpasses the state of the art in multilingual reasoning. To promote reproducibility, we release our corpus, models, and training pipeline under open licenses at hf.co/britllm/CuatroLLM.