Assessing the Role of Data Quality in Training Bilingual Language Models

TL;DR

This paper investigates how data quality impacts the performance of bilingual language models, revealing that filtering high-quality data can significantly improve multilingual NLP outcomes across various languages.

Contribution

It introduces a data filtering strategy that enhances bilingual model performance by prioritizing high-quality data, addressing a key challenge in multilingual NLP.

Findings

Data quality significantly affects bilingual model performance.

Filtering high-quality data reduces performance gaps between languages.

The approach improves monolingual performance by 2-4%.

Abstract

Bilingual and multilingual language models offer a promising path toward scaling NLP systems across diverse languages and users. However, their performance often varies wildly between languages as prior works show that adding more languages can degrade performance for some languages (such as English), while improving others (typically more data constrained languages). In this work, we investigate causes of these inconsistencies by comparing bilingual and monolingual language models. Our analysis reveals that unequal data quality, not just data quantity, is a major driver of performance degradation in bilingual settings. We propose a simple yet effective data filtering strategy to select higher-quality bilingual training data with only high quality English data. Applied to French, German, and Chinese, our approach improves monolingual performance by 2-4% and reduces bilingual model performance gaps to 1%. These results highlight the overlooked importance of data quality in multilingual pretraining and offer a practical recipe for balancing performance.