The Impact of Language Mixing on Bilingual LLM Reasoning

TL;DR

This paper investigates how language switching in bilingual LLMs can enhance reasoning, showing that encouraging language mixing improves accuracy and can be strategically beneficial rather than a training artifact.

Contribution

It identifies reinforcement learning with verifiable rewards as a key factor in language mixing and demonstrates that promoting language switching can improve reasoning performance.

Findings

Language mixing can boost reasoning accuracy by 5.6 percentage points.

A lightweight probe predicts when language switching benefits reasoning.

Guided decoding with the probe increases accuracy by 2.92 percentage points.

Abstract

Proficient multilingual speakers often intentionally switch languages in the middle of a conversation. Similarly, recent reasoning-focused bilingual large language models (LLMs) with strong capabilities in both languages exhibit language mixing-alternating languages within their chain of thought. Discouraging this behavior in DeepSeek-R1 was found to degrade accuracy, suggesting that language mixing may benefit reasoning. In this work, we study language switching in Chinese-English bilingual reasoning models. We identify reinforcement learning with verifiable rewards (RLVR) as the critical training stage that leads to language mixing. We show that language mixing can enhance reasoning: enforcing monolingual decoding reduces accuracy by 5.6 percentage points on MATH500. Additionally, a lightweight probe can be trained to predict whether a potential language switch would benefit or harm reasoning, and when used to guide decoding, increases accuracy by 2.92 percentage points. Our findings suggest that language mixing is not merely a byproduct of multilingual training, but is a strategic reasoning behavior.