Extrapolation by Association: Length Generalization Transfer in Transformers

TL;DR

This paper explores how transformer models can transfer length generalization skills across related tasks, revealing that training on auxiliary tasks enhances their ability to extrapolate to longer inputs in various domains.

Contribution

It demonstrates that length generalization can be transferred between related tasks in transformers, including pretrained models, and links this transfer to attention head reuse.

Findings

Length generalization transfer occurs across diverse tasks.

Pretrained models also exhibit transfer effects.

Attention head reuse correlates with transfer success.

Abstract

Transformer language models have demonstrated impressive generalization capabilities in natural language domains, yet we lack a fine-grained understanding of how such generalization arises. In this paper, we investigate length generalization--the ability to extrapolate from shorter to longer inputs--through the lens of \textit{task association}. We find that length generalization can be \textit{transferred} across related tasks. That is, training a model with a longer and related auxiliary task can lead it to generalize to unseen and longer inputs from some other target task. We demonstrate this length generalization transfer across diverse algorithmic tasks, including arithmetic operations, string transformations, and maze navigation. Our results show that transformer models can inherit generalization capabilities from similar tasks when trained jointly. Moreover, we observe similar transfer effects in pretrained language models, suggesting that pretraining equips models with reusable computational scaffolding that facilitates extrapolation in downstream settings. Finally, we provide initial mechanistic evidence that length generalization transfer correlates with the re-use of the same attention heads between the tasks. Together, our findings deepen our understanding of how transformers generalize to out-of-distribution inputs and highlight the compositional reuse of inductive structure across tasks.