MrRoPE: Mixed-radix Rotary Position Embedding

TL;DR

MrRoPE introduces a unified, theory-based approach to extend Rotary Position Embedding for longer sequences, enabling high-performance, training-free generalization in large-context tasks.

Contribution

It proposes a generalized radix conversion framework for RoPE extensions and introduces two training-free methods, MrRoPE-Uni and MrRoPE-Pro, for effective long-sequence encoding.

Findings

MrRoPE-Pro achieves over 85% recall on 128K-context tasks.

It more than doubles YaRN's accuracy on Infinite-Bench retrieval.

Theoretical analysis confirms increased upper bounds for encoding length.

Abstract

Rotary Position Embedding (RoPE)-extension refers to modifying or generalizing the Rotary Position Embedding scheme to handle longer sequences than those encountered during pre-training. However, current extension strategies are highly diverse and lack a unified theoretical foundation. In this paper, we propose MrRoPE (Mixed-radix RoPE), a generalized encoding formulation based on a radix system conversion perspective, which elegantly unifies various RoPE-extension approaches as distinct radix conversion strategies. Based on this theory, we introduce two training-free extensions, MrRoPE-Uni and MrRoPE-Pro, which leverage uniform and progressive radix conversion strategies, respectively, to achieve 'train short, test long' generalization. Without fine-tuning, MrRoPE-Pro sustains over 85% recall in the 128K-context Needle-in-a-Haystack test and achieves more than double YaRN's accuracy on Infinite-Bench retrieval and dialogue subsets. Theoretical analysis confirms that MrRoPE-Pro effectively raises the upper bound of RoPE's attainable encoding length, which further validates the reliability and utility of our theory and methodology.