$\pi$-Attention: Periodic Sparse Transformers for Efficient Long-Context Modeling

TL;DR

The paper introduces Attention, a periodic sparse Transformer that efficiently models long contexts by combining ring-local neighborhoods, deterministic skips, and adaptive fusion, achieving comparable or better performance with reduced computational costs.

Contribution

It proposes a novel periodic sparse attention mechanism that improves long-range modeling efficiency and effectiveness over existing sparse attention methods.

Findings

Attention achieves (50%) fewer GPUs with similar or better performance.

It attains (8.3%) lower perplexity than RingAttention in language modeling.

The model's design enables predictable coverage and adaptive fusion for long-context tasks.

Abstract

Transformers have revolutionized natural language processing, but their quadratic complexity with respect to sequence length remains a fundamental bottleneck for long-range modeling. While sparse attention mechanisms like RingAttention reduce computational costs by restricting attention to local neighborhoods, they suffer from limited receptive fields and lack of adaptability. We present \PiAttention, a periodic sparse Transformer that factorizes attention into ring-local neighborhoods, deterministic $\pi$-stride skips, and an adaptive fusion gate. The periodic structure provides predictable coverage of distant tokens, while the sparse footprint keeps the per-layer complexity linear in context length. We prove that \PiAttention achieves $\mathcal{O}(kL + \pi \log L)$ receptive field growth compared to $\mathcal{O}(kL)$ for RingAttention, where $k$ is the local window size, $\pi$ is the skip period, and $L$ is the sequence length. Extensive experiments on language modeling, retrieval, and vision-language tasks demonstrate that \PiAttention matches or surpasses dense attention quality with 8.3\% lower perplexity than RingAttention while using 50\% fewer GPUs for the same context length. Our detailed ablations and visualizations reveal the importance of periodic skips, adaptive fusion, and head-level sparsity coordination for efficient long-context modeling.