Tracing Multilingual Representations in LLMs with Cross-Layer Transcoders

TL;DR

This paper investigates how multilingual LLMs internally represent multiple languages, revealing shared features and language-specific decoding mechanisms, and explains performance gaps through detailed mechanistic analysis.

Contribution

It introduces Cross-Layer Transcoders and attribution graphs to analyze multilingual representations, uncovering shared features and language-specific decoding in LLMs.

Findings

Multilingual shared representations involve similar features across languages.

Language-specific decoding emerges in later layers of the model.

Underperformance in non-English languages is linked to weak late-layer features and tokenizer bias.

Abstract

Multilingual Large Language Models (LLMs) can process many languages, yet how they internally represent this diversity remains unclear. Do they form shared multilingual representations with language-specific decoding, and if so, why does performance favor the dominant training language? To address this, we train models on different multilingual mixtures and analyze their internal mechanisms using Cross-Layer Transcoders (CLTs) and Attribution Graphs. Our results reveal multilingual shared representations: the model employs highly similar features across languages, while language-specific decoding emerges in later layers. Training models without English shows identical multilingual shared space structures. Decoding relies partly on a small set of high-frequency features in the final layers, which linearly encode language identity from early layers. Intervening on these features allows one language to be suppressed and another substituted. Finally, to explain non-English failures, we perform a Model-Diffing experiment: underperformance arises from dim late-layer features, weak middle-layer clusters, and tokenizer bias toward English that forces early layers to specialize in word reassembly. Finetuning strengthens these features and their links, improving token assembly and language-specific decoding, providing a mechanistic explanation for multilingual gaps. Our models and CLTs are available at https://huggingface.co/collections/CausalNLP/multilingual-clts and https://huggingface.co/collections/CausalNLP/multilingual-gpt2-models. Our code is available at: https://github.com/abirharrasse/MultilingualCLTs