The Representational Geometry of Number
Zhimin Hu, Lanhao Niu, Sashank Varma

TL;DR
This paper investigates how number concepts are represented in language models, revealing that while task-specific representations occupy distinct subspaces, they preserve a stable relational structure that can be linearly transformed, enabling flexible understanding.
Contribution
It demonstrates that number representations in language models maintain a shared relational structure across tasks, with subspaces transformable via linear mappings, advancing understanding of conceptual representation.
Findings
Number representations preserve stable relational structures across tasks.
Task-specific features like magnitude and parity are encoded along separable linear directions.
Subspaces of different number concepts are largely transformable into each other via linear mappings.
Abstract
A central question in cognitive science is whether conceptual representations converge onto a shared manifold to support generalization, or diverge into orthogonal subspaces to minimize task interference. While prior work has discovered evidence for both, a mechanistic account of how these properties coexist and transform across tasks remains elusive. We propose that representational sharing lies not in the concepts themselves, but in the geometric relations between them. Using number concepts as a testbed and language models as high-dimensional computational substrates, we show that number representations preserve a stable relational structure across tasks. Task-specific representations are embedded in distinct subspaces, with low-level features like magnitude and parity encoded along separable linear directions. Crucially, we find that these subspaces are largely transformable into…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsCognitive and developmental aspects of mathematical skills · Child and Animal Learning Development · Neurobiology of Language and Bilingualism
