Towards Infinite Length Extrapolation: A Unified Approach

TL;DR

This paper introduces a unified framework for positional encoding in large language models, proposes Adaptive Positional Encoding (APE) for better long-range dependency handling, and demonstrates its effectiveness on datasets with extremely long sequences.

Contribution

It presents a unified reinterpretation of positional encoding methods, introduces APE with adaptive frequency modulation, and provides theoretical conditions for infinite-length extrapolation.

Findings

APE enables models to process sequences up to 32,000 words.

The framework unifies and generalizes existing positional encoding methods.

Theoretical analysis guarantees well-defined normalization over unbounded sequences.

Abstract

Large language models (LLMs) have revolutionized natural language processing, but their ability to process long sequences is fundamentally limited by the context window size during training. Existing length extrapolation methods often suffer from performance degradation or computational inefficiencies. We thereby use a unified framework that reinterprets positional encoding methods as a decomposition of the attention score into a multiplicative transformation and an additive bias. This perspective not only subsumes popular approaches such as relative position embeddings and attention-bias moderated approaches but also exposes their inherent limitations in handling long-range dependencies. To address these shortcomings, motivated by our framework, we introduce Adaptive Positional Encoding (APE), which leverages adaptive frequency modulation and an intricately designed decay bias that incorporates linear, logarithmic, and square-root terms. Our theoretical analysis establishes conditions for infinite-context extrapolation, ensuring that the softmax normalization remains well-defined over unbounded sequences while preserving long-distance correlations, entropy boundedness and gradient positional sensitivity. We substantiate our claims with an experimental case study on TinyStories dataset as well as a new synthetic dataset, \emph{Long Tiny Stories} featuring stories up to 32,000 words. Relevant code, dataset and model weights are available at https://anonymous.4open.science/r/Check-2DAD/.