Quantitative Bounds for Length Generalization in Transformers

TL;DR

This paper provides the first quantitative bounds on the training sequence length needed for transformers to generalize to longer inputs, analyzing various settings and verifying the bounds empirically.

Contribution

It offers the first formal bounds on length generalization in transformers, analyzing different configurations and error controls, and empirically validating the theoretical predictions.

Findings

Length generalization occurs when transformer behavior on longer sequences can be simulated by shorter ones.

Quantitative bounds depend on the complexity of the task and transformer architecture.

Empirical results support the theoretical bounds and insights.

Abstract

We study the problem of length generalization (LG) in transformers: the ability of a model trained on shorter sequences to maintain performance when evaluated on much longer, previously unseen inputs. Prior work by Huang et al. (2025) established that transformers eventually achieve length generalization once the training sequence length exceeds some finite threshold, but left open the question of how large it must be. In this work, we provide the first quantitative bounds on the required training length for length generalization to occur. Motivated by previous empirical and theoretical work, we analyze LG in several distinct problem settings: $\ell_\infty$ error control vs. average error control over an input distribution, infinite-precision softmax attention vs. finite-precision attention (which reduces to an argmax) in the transformer, and one- vs. two-layer transformers. In all scenarios, we prove that LG occurs when the internal behavior of the transformer on longer sequences can be "simulated" by its behavior on shorter sequences seen during training. Our bounds give qualitative estimates for the length of training data required for a transformer to generalize, and we verify these insights empirically. These results sharpen our theoretical understanding of the mechanisms underlying extrapolation in transformers, and formalize the intuition that richer training data is required for generalization on more complex tasks.