mmBERT: A Modern Multilingual Encoder with Annealed Language Learning

TL;DR

mmBERT is a multilingual encoder model trained on over 1800 languages, introducing novel training strategies that significantly improve performance on classification and retrieval tasks across resource levels.

Contribution

The paper presents mmBERT, a new multilingual encoder with innovative training techniques, including inverse mask scheduling and phased low-resource language inclusion, enhancing performance across diverse languages.

Findings

mmBERT achieves performance comparable to leading models like o3 and Gemini 2.5 Pro.

Inclusion of low-resource languages during decay phase boosts model performance.

The model outperforms previous models on classification and retrieval tasks for both high and low-resource languages.

Abstract

Encoder-only languages models are frequently used for a variety of standard machine learning tasks, including classification and retrieval. However, there has been a lack of recent research for encoder models, especially with respect to multilingual models. We introduce mmBERT, an encoder-only language model pretrained on 3T tokens of multilingual text in over 1800 languages. To build mmBERT we introduce several novel elements, including an inverse mask ratio schedule and an inverse temperature sampling ratio. We add over 1700 low-resource languages to the data mix only during the decay phase, showing that it boosts performance dramatically and maximizes the gains from the relatively small amount of training data. Despite only including these low-resource languages in the short decay phase we achieve similar classification performance to models like OpenAI's o3 and Google's Gemini 2.5 Pro. Overall, we show that mmBERT significantly outperforms the previous generation of models on classification and retrieval tasks -- on both high and low-resource languages.