Probing for Arithmetic Errors in Language Models

TL;DR

This paper demonstrates that internal activations in language models can be used to detect and correct arithmetic errors, enabling lightweight self-correction mechanisms that improve accuracy.

Contribution

It introduces simple probing techniques to identify arithmetic errors from internal states and shows their effectiveness in guiding model self-correction.

Findings

Probes accurately decode outputs and correct answers from hidden states.

Error detectors achieve over 90% accuracy in predicting correctness.

Probes trained on simple arithmetic generalize to complex chain-of-thought reasoning.

Abstract

We investigate whether internal activations in language models can be used to detect arithmetic errors. Starting with a controlled setting of 3-digit addition, we show that simple probes can accurately decode both the model's predicted output and the correct answer from hidden states, regardless of whether the model's output is correct. Building on this, we train lightweight error detectors that predict model correctness with over 90% accuracy. We then extend our analysis to structured chain-of-thought traces on addition-only GSM8K problems and find that probes trained on simple arithmetic generalize well to this more complex setting, revealing consistent internal representations. Finally, we demonstrate that these probes can guide selective re-prompting of erroneous reasoning steps, improving task accuracy with minimal disruption to correct outputs. Our findings suggest that arithmetic errors can be anticipated from internal activations alone, and that simple probes offer a viable path toward lightweight model self-correction.