Do Transformers Have the Ability for Periodicity Generalization?

TL;DR

This paper investigates the limitations of Transformer-based large language models in generalizing periodic patterns out-of-distribution, introducing a new benchmark and theoretical insights to understand their struggles.

Contribution

It provides a unified algebraic interpretation of periodicity, constructs a controllable benchmark for composite periodicity, and demonstrates Transformers' limited generalization ability in this domain.

Findings

Transformers memorize but do not generalize composite periodicity.

Models struggle with out-of-distribution extrapolation of periodic patterns.

The paper offers theoretical explanations for these limitations.

Abstract

Large language models (LLMs) based on the Transformer have demonstrated strong performance across diverse tasks. However, current models still exhibit substantial limitations in out-of-distribution (OOD) generalization compared with humans. We investigate this gap through periodicity, one of the basic OOD scenarios. Periodicity captures invariance amid variation. Periodicity generalization represents a model's ability to extract periodic patterns from training data and generalize to OOD scenarios. We introduce a unified interpretation of periodicity from the perspective of abstract algebra and reasoning, including both single and composite periodicity, to explain why Transformers struggle to generalize periodicity. Then we construct Coper about composite periodicity, a controllable generative benchmark with two OOD settings, Hollow and Extrapolation. Experiments reveal that periodicity generalization in Transformers is limited, where models can memorize periodic data during training, but cannot generalize to unseen composite periodicity. We release the source code to support future research.