Grow Up and Merge: Scaling Strategies for Efficient Language Adaptation

TL;DR

This paper demonstrates that scaling up pretrained language models enhances their ability to adapt to low-resource languages efficiently, preserves English capabilities, and offers insights into merging language-specific models for multilingual systems.

Contribution

The study shows that larger, upscaled models enable more effective language adaptation and merging, providing a scalable strategy for multilingual model development.

Findings

Upscaled models match or outperform smaller models with more data.

Scaling preserves English performance and reduces catastrophic forgetting.

Merging larger models yields better results than smaller ones, with room for improved methods.

Abstract

Achieving high-performing language models which include medium- and lower-resource languages remains a challenge. Massively multilingual models still underperform compared to language-specific adaptations, especially at smaller model scales. In this work, we investigate scaling as an efficient strategy for adapting pretrained models to new target languages. Through comprehensive scaling ablations with approximately FLOP-matched models, we test whether upscaling an English base model enables more effective and resource-efficient adaptation than standard continued pretraining. We find that, once exposed to sufficient target-language data, larger upscaled models can match or surpass the performance of smaller models continually pretrained on much more data, demonstrating the benefits of scaling for data efficiency. Scaling also helps preserve the base model's capabilities in English, thus reducing catastrophic forgetting. Finally, we explore whether such scaled, language-specific models can be merged to construct modular and flexible multilingual systems. We find that while merging remains less effective than joint multilingual training, upscaled merges perform better than smaller ones. We observe large performance differences across merging methods, suggesting potential for improvement through merging approaches specialized for language-level integration.