Rethinking what Matters: Effective and Robust Multilingual Realignment for Low-Resource Languages

TL;DR

This paper investigates how strategic language subset selection in realignment can enhance cross-lingual transfer for low-resource languages, reducing data needs while maintaining or improving performance.

Contribution

It demonstrates that carefully chosen, linguistically diverse language subsets can match or outperform full multilingual alignment, optimizing realignment for low-resource languages.

Findings

Selective language subsets can match full alignment performance.

Realignment benefits are more pronounced for low-resource languages.

Informed language selection improves efficiency and robustness.

Abstract

Realignment is a promising strategy to improve cross-lingual transfer in multilingual language models. However, empirical results are mixed and often unreliable, particularly for typologically distant or low-resource languages (LRLs) compared to English. Moreover, word realignment tools often rely on high-quality parallel data, which can be scarce or noisy for many LRLs. In this work, we conduct an extensive empirical study to investigate whether realignment truly benefits from using all available languages, or if strategically selected subsets can offer comparable or even improved cross-lingual transfer, and study the impact on LRLs. Our controlled experiments show that realignment can be particularly effective for LRLs and that using carefully selected, linguistically diverse subsets can match full multilingual alignment, and even outperform it for unseen LRLs. This indicates that effective realignment does not require exhaustive language coverage and can reduce data collection overhead, while remaining both efficient and robust when guided by informed language selection.