One Model to Translate Them All? A Journey to Mount Doom for Multilingual Model Merging

TL;DR

This paper investigates why weight-space model merging often fails in multilingual machine translation, revealing that language-specific neurons and representational divergence hinder effective merging.

Contribution

It provides a systematic analysis of weight-space merging in multilingual models, identifying how fine-tuning redistributes neurons and increases divergence across languages.

Findings

Merging degrades performance more with diverse target languages.

Language-specific neurons are concentrated in embedding and upper transformer layers.

Fine-tuning redistributes neurons, increasing divergence and reducing merging effectiveness.

Abstract

Weight-space model merging combines independently fine-tuned models without accessing original training data, offering a practical alternative to joint training. While merging succeeds in multitask settings, its behavior in multilingual contexts remains poorly understood. We systematically study weight-space merging for multilingual machine translation by fully fine-tuning language model on large-scale bilingual corpora and evaluating standard merging strategies. Our experiments reveal that merging degrades performance, especially when target languages differ. To explain this failure, we analyze internal representations using span-conditioned neuron selectivity and layer-wise centered kernel alignment. We find that language-specific neurons concentrate in embedding layers and upper transformer blocks, while intermediate layers remain largely shared across languages. Critically, fine-tuning redistributes rather than sharpens language selectivity: neurons for supervised and related languages become less exclusive, while those for unsupervised languages grow more isolated. This redistribution increases representational divergence in higher layers that govern generation. These findings suggest that multilingual fine-tuning may reshape geometry in ways that reduce compatibility with standard weight-space merging assumptions. Our work thus provides an explanation for why merging fails in multilingual translation scenarios.