Number Representations in LLMs: A Computational Parallel to Human Perception

TL;DR

This paper investigates whether large language models encode numerical information in a logarithmic, human-like manner, revealing that their internal representations exhibit sublinear spacing similar to human perception.

Contribution

The study demonstrates that LLMs encode numbers in a compressed, non-uniform way, aligning with the logarithmic mental number line hypothesis from cognitive science.

Findings

LLMs show sublinear spacing in numerical representations

Numerical embeddings align with a logarithmic scale

Models encode numbers in a human-like perceptual manner

Abstract

Humans are believed to perceive numbers on a logarithmic mental number line, where smaller values are represented with greater resolution than larger ones. This cognitive bias, supported by neuroscience and behavioral studies, suggests that numerical magnitudes are processed in a sublinear fashion rather than on a uniform linear scale. Inspired by this hypothesis, we investigate whether large language models (LLMs) exhibit a similar logarithmic-like structure in their internal numerical representations. By analyzing how numerical values are encoded across different layers of LLMs, we apply dimensionality reduction techniques such as PCA and PLS followed by geometric regression to uncover latent structures in the learned embeddings. Our findings reveal that the model's numerical representations exhibit sublinear spacing, with distances between values aligning with a logarithmic scale. This suggests that LLMs, much like humans, may encode numbers in a compressed, non-uniform manner.