Layer Specialization Underlying Compositional Reasoning in Transformers

TL;DR

This paper investigates how transformers develop layered, specialized representations that enable compositional reasoning, using a probabilistic grammar model and analyzing training dynamics and internal structures.

Contribution

It reveals the progressive emergence of layer specialization in transformers that correlates with their ability to generalize in compositional tasks.

Findings

Layer specialization emerges gradually during training.

Transformers develop hierarchical, structured internal representations.

Performance improves with task complexity and number of in-context examples.

Abstract

Transformers exhibit compositional reasoning on sequences not observed during training, a capability often attributed to in-context learning (ICL) and skill composition. We investigate this phenomenon using the Random Hierarchy Model (RHM), a probabilistic context-free grammar that generates sequences through recursive rule application. Models are trained on subsets of sequences and evaluated across four generalization conditions: memorization, in-distribution generalization, out-of-distribution generalization with the same rules, and cross-layer transfer. Behaviorally, performance improves systematically with task complexity and the number of in-context examples, with out-of-distribution tasks requiring substantially more examples than in-distribution scenarios. Mechanistically, we identify a progressive emergence of layer specialization during training that correlates with generalization performance. Principal component analysis and attention pattern clustering reveal that transformers develop structured, hierarchically organized representations in specialized layers. These results demonstrate that transformers develop modular, interpretable mechanisms supporting compositional reasoning, linking internal algorithmic structure to observed behavioral capabilities.