Bridging Latent Reasoning and Target-Language Generation via Retrieval-Transition Heads

TL;DR

This paper investigates retrieval and retrieval-transition heads in multilingual transformers, revealing their roles in language-specific output and reasoning, and demonstrating their importance through extensive multilingual benchmark evaluations.

Contribution

It identifies and characterizes Retrieval-Transition Heads (RTH) as distinct from retrieval heads, crucial for target-language generation and reasoning in multilingual models.

Findings

RTHs are shared across multiple languages.

Masking RTHs causes larger performance drops than masking retrieval heads.

RTHs are vital for Chain-of-Thought reasoning in multilingual LLMs.

Abstract

Recent work has identified a subset of attention heads in Transformer as retrieval heads, which are responsible for retrieving information from the context. In this work, we first investigate retrieval heads in multilingual contexts. In multilingual language models, we find that retrieval heads are often shared across multiple languages. Expanding the study to cross-lingual setting, we identify Retrieval-Transition heads(RTH), which govern the transition to specific target-language output. Our experiments reveal that RTHs are distinct from retrieval heads and more vital for Chain-of-Thought reasoning in multilingual LLMs. Across four multilingual benchmarks (MMLU-ProX, MGSM, MLQA, and XQuaD) and two model families (Qwen-2.5 and Llama-3.1), we demonstrate that masking RTH induces bigger performance drop than masking Retrieval Heads (RH). Our work advances understanding of multilingual LMs by isolating the attention heads responsible for mapping to target languages.