Symmetry in language statistics shapes the geometry of model representations

TL;DR

This paper demonstrates that the geometric structure of language model representations is governed by translation symmetry in language statistics, with theoretical proofs and empirical validation showing robustness and universality of these geometric manifolds.

Contribution

The paper provides a theoretical framework linking language statistics symmetry to the geometry of model representations, supported by empirical evidence and robustness analysis.

Findings

Language statistics exhibit translation symmetry.

Model representations form predictable geometric manifolds.

Robustness of geometry persists under data perturbations.

Abstract

The internal representations learned by language models consistently exhibit striking geometric structure: calendar months organize into a circle, historical years form a smooth one-dimensional manifold, and cities' latitudes and longitudes can be decoded using a linear probe. To explain this neural code, we first show that language statistics exhibit translation symmetry (for example, the frequency with which any two months co-occur in text depends only on the time interval between them). We prove that this symmetry governs these geometric structures in high-dimensional word embedding models, and we analytically derive the manifold geometry of word representations. These predictions empirically match large text embedding models and large language models. Moreover, the representational geometry persists at moderate embedding dimension even when the relevant statistics are perturbed (e.g., by removing all sentences in which two months co-occur). We prove that this robustness emerges naturally when the co-occurrence statistics are controlled by an underlying latent variable. These results suggest that representational manifolds have a universal origin: symmetry in the statistics of natural data.