Remember to Forget: Gated Adaptive Positional Encoding

TL;DR

GAPE is a novel positional encoding method that enhances long-context robustness in language models by introducing content-aware, gate-based attention modulation while maintaining rotary geometry.

Contribution

It proposes GAPE, a drop-in augmentation for rotary positional encoding that improves long-range attention stability without sacrificing local resolution.

Findings

GAPE yields sharper attention compared to rotary baselines.

GAPE improves robustness in synthetic retrieval and long-context benchmarks.

GAPE can be implemented within standard scaled dot-product attention.

Abstract

Rotary Positional Encoding (RoPE) is widely used in modern large language models. However, when sequences are extended beyond the range seen during training, rotary phases can enter out-of-distribution regimes, leading to spurious long-range alignments, diffuse attention, and degraded retrieval. Existing remedies only partially address these failures, as they often trade local positional resolution for long-context stability. We propose GAPE (Gated Adaptive Positional Encoding), a drop-in augmentation for positional encodings that introduces a content-aware bias directly into the attention logits while preserving the rotary geometry. GAPE decouples distance-based suppression from token importance through a query-dependent gate that contracts irrelevant context and a key-dependent gate that preserves salient distant tokens. We prove that protected tokens remain accessible, while the attention mass assigned to unprotected distant tokens decays as a function of the query gate. We further show that GAPE can be implemented within standard scaled dot-product attention. We validate these properties empirically, finding that GAPE consistently yields sharper attention and improved long-context robustness over rotary baselines across both synthetic retrieval and long-context benchmarks.