Weber's Law in Transformer Magnitude Representations: Efficient Coding, Representational Geometry, and Psychophysical Laws in Language Models

TL;DR

This study investigates how transformer language models represent magnitude, revealing consistent log-compressive geometry that is dissociated from behavioral performance, and shows that training data statistics induce this geometry without necessarily leading to behavioral competence.

Contribution

The paper provides the first comprehensive psychophysical analysis of magnitude representations in transformers, demonstrating the prevalence of Weber-law consistent geometry and its dissociation from behavior.

Findings

Representational geometry is consistently log-compressive across models.

Geometry is dissociated from behavioral discrimination performance.

Early layers are causally involved in magnitude processing, later layers are not.

Abstract

How do transformer language models represent magnitude? Recent work disagrees: some find logarithmic spacing, others linear encoding, others per-digit circular representations. We apply the formal tools of psychophysics to resolve this. Using four converging paradigms (representational similarity analysis, behavioural discrimination, precision gradients, causal intervention) across three magnitude domains in three 7-9B instruction-tuned models spanning three architecture families (Llama, Mistral, Qwen), we report three findings. First, representational geometry is consistently log-compressive: RSA correlations with a Weber-law dissimilarity matrix ranged from .68 to .96 across all 96 model-domain-layer cells, with linear geometry never preferred. Second, this geometry is dissociated from behaviour: one model produces a human-range Weber fraction (WF = 0.20) while the other does not, and both models perform at chance on temporal and spatial discrimination despite possessing logarithmic geometry. Third, causal intervention reveals a layer dissociation: early layers are functionally implicated in magnitude processing (4.1x specificity) while later layers where geometry is strongest are not causally engaged (1.2x). Corpus analysis confirms the efficient coding precondition (alpha = 0.77). These results suggest that training data statistics alone are sufficient to produce log-compressive magnitude geometry, but geometry alone does not guarantee behavioural competence.