Repeated-Token Counting Reveals a Dissociation Between Representations and Outputs

TL;DR

Large language models accurately encode token counts internally, but a format-specific MLP block overwrites this information, causing counting failures despite correct internal representations.

Contribution

This paper demonstrates that counting failures are due to a format-triggered MLP overwriting correct counts, not a lack of internal representation.

Findings

Linear probes decode correct counts at all layers.

A format-triggered MLP overwrites counts in certain formats.

Counting failures are due to routing issues, not representation.

Abstract

Large language models fail at counting repeated tokens despite strong performance on broader reasoning benchmarks. These failures are commonly attributed to limitations in internal count tracking. We show this attribution is wrong. Linear probes on the residual stream decode the correct count with near-perfect accuracy at every post-embedding layer, across all model depths. This holds even at the exact layers where the wrong answer crystallizes while the model simultaneously outputs an incorrect count. Attention patterns show no evidence of collapse over repeated tokens and tokenization artifacts account for none of the failure. Instead, a format-triggered multi-layer perceptron (MLP) block overwrites the correctly-encoded count with a fixed wrong answer at roughly 88--93,% network depth. This prior fires for repeated word-tokens in space-separated list format and is absent for repeated digit-tokens. It is suppressed by comma-separated delimiters in larger models but persists in smaller ones. The finding holds across Llama-3.2 (1B and 3B) and Qwen2.5 (1.5B, 3B and 7B) at consistent relative depth. Counting failure is a failure of routing not of representation and the two require different interventions.