LLMs Know More About Numbers than They Can Say

TL;DR

This paper investigates whether language models truly understand numerical magnitudes by probing their hidden states, revealing they encode number sizes well internally but struggle with explicit ranking tasks, and shows finetuning can improve their numerical reasoning.

Contribution

It demonstrates that LLMs internally encode numerical magnitudes effectively and introduces a method to improve their explicit numerical ranking abilities through auxiliary training.

Findings

Hidden states encode log-magnitudes with about 2.3% relative error.

Hidden states can rank numerals with over 90% accuracy.

Finetuning with a magnitude-based auxiliary loss improves ranking accuracy by 3.22%.

Abstract

Although state-of-the-art LLMs can solve math problems, we find that they make errors on numerical comparisons with mixed notation: "Which is larger, $5.7 \times 10^2$ or $580$?" This raises a fundamental question: Do LLMs even know how big these numbers are? We probe the hidden states of several smaller open-source LLMs. A single linear projection of an appropriate hidden layer encodes the log-magnitudes of both kinds of numerals, allowing us to recover the numbers with relative error of about 2.3% (on restricted synthetic text) or 19.06% (on scientific papers). Furthermore, the hidden state after reading a pair of numerals encodes their ranking, with a linear classifier achieving over 90% accuracy. Yet surprisingly, when explicitly asked to rank the same pairs of numerals, these LLMs achieve only 50-70% accuracy, with worse performance for models whose probes are less effective. Finally, we show that incorporating the classifier probe's log-loss as an auxiliary objective during finetuning brings an additional 3.22% improvement in verbalized accuracy over base models, demonstrating that improving models' internal magnitude representations can enhance their numerical reasoning capabilities. Our code is available at https://github.com/VCY019/Numeracy-Probing.