Shattered Compositionality: Counterintuitive Learning Dynamics of Transformers for Arithmetic

TL;DR

This paper investigates how transformers learn arithmetic skills, revealing they often acquire skills in non-human-like ways, leading to errors, especially under distribution shifts, due to reliance on correlational rather than causal learning mechanisms.

Contribution

It uncovers the phenomenon of shattered compositionality in transformers, showing their learning dynamics differ from human-like sequential skill acquisition, and provides insights into underlying causes.

Findings

Transformers often learn skills in reverse order or in parallel.

Shattered compositionality persists across modern LLMs and is unaffected by scaling.

Correlational matching to training data influences learning dynamics.

Abstract

Large language models (LLMs) often exhibit unexpected errors or unintended behavior, even at scale. While recent work reveals the discrepancy between LLMs and humans in skill compositions, the learning dynamics of skill compositions and the underlying cause of non-human behavior remain elusive. In this study, we investigate the mechanism of learning dynamics by training transformers on synthetic arithmetic tasks. Through extensive ablations and fine-grained diagnostic metrics, we discover that transformers do not reliably build skill compositions according to human-like sequential rules. Instead, they often acquire skills in reverse order or in parallel, which leads to unexpected mixing errors especially under distribution shifts--a phenomenon we refer to as shattered compositionality. To explain these behaviors, we provide evidence that correlational matching to the training data, rather than causal or procedural composition, shapes learning dynamics. We further show that shattered compositionality persists in modern LLMs and is not mitigated by pure model scaling or scratchpad-based reasoning. Our results reveal a fundamental mismatch between a model's learning behavior and desired skill compositions, with implications for reasoning reliability, out-of-distribution robustness, and alignment.