Multilingual Language Models Encode Script Over Linguistic Structure

TL;DR

This study reveals that multilingual language models primarily organize their internal representations based on orthographic surface forms rather than abstract linguistic structures, with abstraction emerging gradually in deeper layers.

Contribution

The paper introduces a novel analysis of multilingual LMs showing their representations are strongly influenced by orthography, and typological abstraction develops progressively across layers.

Findings

Orthography strongly conditions language-associated units.

Romanization causes near-disjoint representations across scripts.

Deeper layers reveal more accessible typological structures.

Abstract

Multilingual language models (LMs) organize representations for typologically and orthographically diverse languages into a shared parameter space, yet the nature of this internal organization remains elusive. In this work, we investigate which linguistic properties - abstract language identity or surface-form cues - shape multilingual representations. To do so, we analyze language-associated units across different model families and scales using the Language Activation Probability Entropy (LAPE) metric, and further decompose activations with Sparse Autoencoders. We find that these units are strongly conditioned on orthography: romanization induces near-disjoint representations that align with neither native-script inputs nor English, while word-order shuffling has limited effect on unit identity. Probing shows that typological structure becomes increasingly accessible in deeper layers, while causal interventions indicate that generation is most sensitive to units that are invariant to surface-form perturbations rather than to units identified by typological alignment alone. Overall, our results suggest that multilingual LMs organize representations around surface form, with linguistic abstraction emerging gradually without collapsing into a unified interlingua.