Controlling Language Confusion in Multilingual LLMs

TL;DR

This paper addresses language confusion in multilingual large language models by introducing a penalty-based method called ORPO, which reduces unintended language mixing without sacrificing overall performance, especially benefiting low-resource settings.

Contribution

The paper proposes ORPO, a novel penalty-based approach that explicitly discourages language confusion in multilingual models, improving language consistency during generation.

Findings

ORPO effectively reduces language mixing in multilingual LLMs.

Models maintain strong performance on general tasks with ORPO.

Language confusion is mitigated even at high decoding temperatures.

Abstract

Large language models often suffer from language confusion, a phenomenon in which responses are partially or entirely generated in unintended languages. This critically degrades the user experience, especially in low-resource settings. We hypothesize that this issue stems from limitations in conventional fine-tuning objectives, such as supervised learning, which optimize the likelihood of correct tokens without explicitly penalizing undesired outputs such as cross-lingual mixing. Analysis of loss trajectories during pretraining further reveals that models fail to distinguish between monolingual and language-mixed texts, highlighting the absence of inherent pressure to avoid such confusion. In this work, we apply ORPO, which adds penalties for unwanted output styles to standard SFT, effectively suppressing language-confused generations. ORPO maintains strong language consistency, even under high decoding temperatures, while preserving general QA performance. Our findings suggest that incorporating appropriate penalty terms can effectively mitigate language confusion in multilingual models, particularly in low-resource scenarios.