LaMPE: Length-aware Multi-grained Positional Encoding for Adaptive Long-context Scaling Without Training

TL;DR

LaMPE is a training-free, length-aware positional encoding method that adaptively allocates positional capacity in RoPE-based LLMs, significantly improving long-context performance without retraining.

Contribution

Introduces LaMPE, a novel dynamic, multi-grained positional encoding approach that enhances long-context scaling in LLMs without additional training.

Findings

Significant performance gains on long-context benchmarks.

Effective adaptation to varying input lengths.

Compatibility with existing RoPE-based LLMs.

Abstract

Large language models (LLMs) experience significant performance degradation when the input exceeds the pretraining context window, primarily due to the out-of-distribution (OOD) behavior of Rotary Position Embedding (RoPE). Recent studies mitigate this problem by remapping OOD positions into the in-distribution range with fixed mapping strategies, ignoring the dynamic relationship between input length and the model's effective context window. To this end, we propose Length-aware Multi-grained Positional Encoding (LaMPE), a training-free method that fully utilizes the model's effective context window for adaptive long-context scaling in LLMs. Motivated by the left-skewed frequency distribution of relative positions, LaMPE establishes a dynamic relationship between mapping length and input length through a parametric scaled sigmoid function to adaptively allocate positional capacity across varying input lengths. Meanwhile, LaMPE devises a novel multi-grained attention mechanism that strategically allocates positional resolution across different sequence regions to capture both fine-grained locality and long-range dependencies. Our method can be seamlessly applied to a wide range of RoPE-based LLMs without training. Extensive experiments on three representative LLMs across five mainstream long-context benchmarks demonstrate that LaMPE achieves significant performance improvements compared to existing length extrapolation methods. The code will be released at https://github.com/scar-on/LaMPE.