How Language Directions Align with Token Geometry in Multilingual LLMs

TL;DR

This paper systematically analyzes how multilingual large language models encode language information across their layers, revealing that language separation occurs early and is influenced by training data composition, with implications for fairness and data strategies.

Contribution

It introduces a comprehensive probing framework and a new Token–Language Alignment analysis to understand language encoding in multilingual LLMs, highlighting the impact of training data on language representation.

Findings

Language information becomes sharply separated in early layers.

Language directions are strongly aligned with training data composition.

Chinese-inclusive models show higher language alignment than English-centric models.

Abstract

Multilingual LLMs demonstrate strong performance across diverse languages, yet there has been limited systematic analysis of how language information is structured within their internal representation space and how it emerges across layers. We conduct a comprehensive probing study on six multilingual LLMs, covering all 268 transformer layers, using linear and nonlinear probes together with a new Token--Language Alignment analysis to quantify the layer-wise dynamics and geometric structure of language encoding. Our results show that language information becomes sharply separated in the first transformer block (+76.4$\pm$8.2 percentage points from Layer 0 to 1) and remains almost fully linearly separable throughout model depth. We further find that the alignment between language directions and vocabulary embeddings is strongly tied to the language composition of the training data. Notably, Chinese-inclusive models achieve a ZH Match@Peak of 16.43\%, whereas English-centric models achieve only 3.90\%, revealing a 4.21$\times$ structural imprinting effect. These findings indicate that multilingual LLMs distinguish languages not by surface script features but by latent representational structures shaped by the training corpus. Our analysis provides practical insights for data composition strategies and fairness in multilingual representation learning. All code and analysis scripts are publicly available at: https://github.com/thisiskorea/How-Language-Directions-Align-with-Token-Geometry-in-Multilingual-LLMs.