Understanding and Improving Length Generalization in Hierarchical Sparse Attention Models

TL;DR

This paper dissects chunk-based sparse attention models to identify key architectural principles that enable effective length generalization in language models, achieving state-of-the-art results in extreme context extrapolation.

Contribution

It systematically uncovers three core design principles crucial for length generalization in sparse attention models, guiding future development of long-context language models.

Findings

Identified three critical architectural components for length generalization.

Achieved state-of-the-art length extrapolation from 4K to 32 million tokens.

Provided theoretical insights into intra-chunk information processing.

Abstract

Effectively processing long contexts is a critical challenge for language models. While standard Transformers are limited by quadratic complexity and poor length extrapolation, alternative architectures like sliding window attention and state space models sacrifice the ability to effectively utilize the full context due to their fixed-size memory. Chunk-based sparse attention has emerged as a promising paradigm for extreme length generalization, yet the key architectural principles underpinning its success are not yet fully understood. In this work, we present a systematic dissection of these models to identify the core components driving their performance. Through a unified framework and comprehensive ablation studies, we demonstrate that a combination of three design principles is critical: (1) an expressive, non-linear Chunk Encoder with a dedicated CLS token to produce representations for retrieval; (2) a Bypassing Residual Path to stably integrate retrieved global information without it being overridden by the local residual stream; and (3) enforced selection sparsity during pre-training to bridge the train-test distribution gap. We provide a theoretical motivation for intra-chunk information processing and landmark generation. By combining these principles, we establish a new state-of-the-art for training-free length extrapolation, successfully generalizing models trained on a 4K context to 32 million tokens on RULER and BABILong. Our findings provide a clear and empirically-grounded set of design principles for developing future, highly-capable long-context language models.