LangGPS: Language Separability Guided Data Pre-Selection for Joint Multilingual Instruction Tuning

TL;DR

This paper introduces LangGPS, a two-stage data pre-selection framework guided by language separability, which enhances multilingual instruction tuning by selecting data that better reflects linguistic distinctions, improving model performance especially for low-resource languages.

Contribution

The paper proposes a novel language separability-guided pre-selection method that improves data selection for multilingual training, leading to better model generalization and understanding across languages.

Findings

LangGPS improves effectiveness of existing data selection methods.

Highly separable samples aid in clearer language boundary formation.

Language separability signals benefit curriculum learning in multilingual training.

Abstract

Joint multilingual instruction tuning is a widely adopted approach to improve the multilingual instruction-following ability and downstream performance of large language models (LLMs), but the resulting multilingual capability remains highly sensitive to the composition and selection of the training data. Existing selection methods, often based on features like text quality, diversity, or task relevance, typically overlook the intrinsic linguistic structure of multilingual data. In this paper, we propose LangGPS, a lightweight two-stage pre-selection framework guided by language separability which quantifies how well samples in different languages can be distinguished in the model's representation space. LangGPS first filters training data based on separability scores and then refines the subset using existing selection methods. Extensive experiments across six benchmarks and 22 languages demonstrate that applying LangGPS on top of existing selection methods improves their effectiveness and generalizability in multilingual training, especially for understanding tasks and low-resource languages. Further analysis reveals that highly separable samples facilitate the formation of clearer language boundaries and support faster adaptation, while low-separability samples tend to function as bridges for cross-lingual alignment. Besides, we also find that language separability can serve as an effective signal for multilingual curriculum learning, where interleaving samples with diverse separability levels yields stable and generalizable gains. Together, we hope our work offers a new perspective on data utility in multilingual contexts and support the development of more linguistically informed LLMs.