Language Models Encode Numbers Using Digit Representations in Base 10

TL;DR

This paper reveals that large language models encode numbers as individual digits in base 10, explaining their errors in numerical reasoning and providing insights into their internal representations.

Contribution

It demonstrates that LLMs use digit-wise representations rather than numeric values, offering a new perspective on how models process and reason with numbers.

Findings

LLMs represent numbers with separate digit encodings in base 10

Errors in numerical tasks are linked to digit-wise representations

Digit representations influence models' numerical reasoning errors

Abstract

Large language models (LLMs) frequently make errors when handling even simple numerical problems, such as comparing two small numbers. A natural hypothesis is that these errors stem from how LLMs represent numbers, and specifically, whether their representations of numbers capture their numeric values. We tackle this question from the observation that LLM errors on numerical tasks are often distributed across the digits of the answer rather than normally around its numeric value. Through a series of probing experiments and causal interventions, we show that LLMs internally represent numbers with individual circular representations per-digit in base 10. This digit-wise representation, as opposed to a value representation, sheds light on the error patterns of models on tasks involving numerical reasoning and could serve as a basis for future studies on analyzing numerical mechanisms in LLMs.