Cross-lingual Collapse: How Language-Centric Foundation Models Shape Reasoning in Large Language Models

TL;DR

This paper investigates how multilingual large language models tend to revert to their dominant language during reasoning tasks, revealing a trade-off between reasoning depth and language fidelity, and explores methods to mitigate this issue.

Contribution

It formalizes the phenomenon of Cross-lingual Collapse in LLMs, systematically studies its causes, and proposes interventions to reduce language drift during reasoning.

Findings

CoT reasoning drifts toward dominant language as performance improves

Language drift is amplified by priors, task difficulty, and decoding strategies

Interventions can mitigate collapse but introduce a performance-fidelity trade-off

Abstract

Reinforcement learning with verifiable reward (RLVR) has been instrumental in eliciting strong reasoning capabilities from large language models (LLMs) via long chains of thought (CoT). During RLVR training, we formalize and systemically study an empirical phenomenon whereby a multilingual model's CoT reverts to its dominant pre-training language (e.g., English) even when prompted in another language, which we term Cross-lingual Collapse. Because the long-CoT regime magnifies exposure to linguistic priors, the underlying trade-off between maximizing reasoning depth and preserving target-language fidelity has remained under-characterized. To examine this trade-off, we train LLMs with Group-Relative Policy Optimization (GRPO) on translated versions of math datasets widely used to elicit long-CoT reasoning. Throughout training, we track both task accuracy and the language consistency of reasoning chains. Our experiments yield three findings: (i) under RLVR, CoT in LLMs systematically drifts toward the pre-training dominant language as reasoning performance rises; (ii) English-centric priors, long-CoT GRPO optimization, task difficulty, and high-entropy decoding jointly amplify this drift, and the pattern persists beyond mathematics; and (iii) interventions that favor target-language traces--via a language-consistency reward, decoding-time controls, or more balanced backbones--mitigate collapse but reveal a persistent performance-fidelity trade-off.