Born a Transformer -- Always a Transformer? On the Effect of Pretraining on Architectural Abilities

TL;DR

This paper investigates how large-scale pretraining affects the inherent architectural limitations of transformers, revealing that pretraining improves some capabilities but does not fully overcome fundamental length-generalization constraints.

Contribution

The study provides a theoretical and empirical analysis of how pretraining influences transformer abilities, especially in length generalization and retrieval tasks, highlighting persistent limitations.

Findings

Pretrained models show better right-side token retrieval (induction) than left-side (anti-induction).

Targeted fine-tuning can eliminate the induction-anti-induction asymmetry.

Pretraining enhances certain capabilities but does not fully overcome length-generalization limits.

Abstract

Transformers have theoretical limitations in modeling certain sequence-to-sequence tasks, yet it remains largely unclear if these limitations play a role in large-scale pretrained LLMs, or whether LLMs might effectively overcome these constraints in practice due to the scale of both the models themselves and their pretraining data. We explore how these architectural constraints manifest after pretraining, by studying a family of $\textit{retrieval}$ and $\textit{copying}$ tasks inspired by Liu et al. [2024a]. We use a recently proposed framework for studying length generalization [Huang et al., 2025] to provide guarantees for each of our settings. Empirically, we observe an $\textit{induction-versus-anti-induction}$ asymmetry, where pretrained models are better at retrieving tokens to the right (induction) rather than the left (anti-induction) of a query token. This asymmetry disappears upon targeted fine-tuning if length-generalization is guaranteed by theory. Mechanistic analysis reveals that this asymmetry is connected to the differences in the strength of induction versus anti-induction circuits within pretrained transformers. We validate our findings through practical experiments on real-world tasks demonstrating reliability risks. Our results highlight that pretraining selectively enhances certain transformer capabilities, but does not overcome fundamental length-generalization limits.