Addition in Four Movements: Mapping Layer-wise Information Trajectories in LLMs

TL;DR

This paper investigates the internal computational process of large language models during multi-digit addition, revealing a hierarchical, four-stage information trajectory that mirrors human problem-solving steps.

Contribution

It introduces a novel four-stage framework for mapping layer-wise information flow in LLMs during addition, combining linear probing and logit-lens techniques.

Findings

Decodable formula-structure representations emerge early.

Numerical abstractions become clearer in deeper layers.

Final output tokens are reliably ranked at the top.

Abstract

Multi-digit addition is a clear probe of the computational power of large language models. To dissect the internal arithmetic processes in LLaMA-3-8B-Instruct, we combine linear probing with logit-lens inspection. Inspired by the step-by-step manner in which humans perform addition, we propose and analyze a coherent four-stage trajectory in the forward pass:Formula-structure representations become linearly decodable first, while the answer token is still far down the candidate list.Core computational features then emerge prominently.At deeper activation layers, numerical abstractions of the result become clearer, enabling near-perfect detection and decoding of the individual digits in the sum.Near the output, the model organizes and generates the final content, with the correct token reliably occupying the top rank.This trajectory suggests a hierarchical process that favors internal computation over rote memorization. We release our code and data to facilitate reproducibility.