Arithmetic OOD Failure Unfolds in Stages in Minimal GPTs

TL;DR

This paper dissects the stages of out-of-distribution failures in minimal GPT models trained on 2-digit addition, revealing layout, carry, recomposition, and residual challenges.

Contribution

It provides a detailed, stage-wise analysis of arithmetic out-of-distribution failures in minimal GPTs, highlighting specific bottlenecks and interventions.

Findings

Layout barrier caused by absolute-position model under layout shift

Carry semantics can be reversed with targeted probes after layout repair

Recomposition bottleneck persists, but targeted data improves performance

Abstract

Arithmetic benchmarks are often reduced to a single held-out score, but that score can conflate qualitatively different failures. We study a controlled minimal GPT trained on exhaustive 2-digit addition, where all local digit transitions are already present in training, and ask why 3-digit generalization still fails. The failure is staged. First, there is a layout barrier: a learned absolute-position model collapses under a pure 3-digit layout shift, and mixed-layout exposure is the only intervention that materially weakens this barrier. Second, after layout repair, the hundreds position behaves like a carry flag rather than a semantic hundreds digit; targeted carry probes reverse the relevant logit margin, whereas a matched extra-data control does not. Third, after carry repair, the main remaining bottleneck is conditional recomposition: high-conditioned tail data outperforms a matched control, high-only data, and tail-only data on all true-3-digit suites, and the same ordering reappears in a larger 2-layer bridge experiment. The residual errors after recomposition are then overwhelmingly tens-only, and a separate 10-seed late-stage study shows that a sign-aware tens repair raises exact match on the hardest thousands-carry suite from 0.664 to 0.822. We therefore provide an experimentally testable decomposition of arithmetic OOD failure into layout, carry-semantics, recomposition, and late tens-residual stages.