Merge and Conquer: Instructing Multilingual Models by Adding Target Language Weights

TL;DR

This paper investigates merging instruction-tuned multilingual models with language-specific base models to improve low-resource language performance efficiently, reducing the need for extensive retraining.

Contribution

It introduces a systematic exploration of model merging as a lightweight alternative for adapting multilingual models to low-resource languages, demonstrating its effectiveness across several Iberian languages.

Findings

Merging enables instruction following in new languages without additional fine-tuning.

The approach supports multilingual capabilities by combining multiple language-specific models.

Model merging achieves competitive performance with significantly lower computational costs.

Abstract

Large Language Models (LLMs) remain heavily centered on English, with limited performance in low-resource languages. Existing adaptation approaches, such as continual pre-training, demand significant computational resources. In the case of instructed models, high-quality instruction data is also required, both of which are often inaccessible for low-resource language communities. Under these constraints, model merging offers a lightweight alternative, but its potential in low-resource contexts has not been systematically explored. In this work, we explore whether it is possible to transfer language knowledge to an instruction-tuned LLM by merging it with a language-specific base model, thereby eliminating the need of language-specific instructions and repeated fine-tuning processes whenever stronger instructed variants become available. Through experiments covering four Iberian languages (Basque, Catalan, Galician, and Spanish) and two model families, we show that merging enables effective instruction following behavior in new languages and even supports multilingual capability through the combination of multiple language-specific models. Our results indicate that model merging is a viable and efficient alternative to traditional adaptation methods for low-resource languages, achieving competitive performance while greatly reducing computational cost.