Fractional Rotation, Full Potential? Investigating Performance and Convergence of Partial RoPE

TL;DR

This paper systematically investigates the effects of applying Rotary Positional Embedding (RoPE) to only a fraction of transformer dimensions, revealing that minimal RoPE usage can save memory while maintaining comparable performance and offering practical training insights.

Contribution

It provides the first comprehensive analysis of partial RoPE, demonstrating that applying RoPE to about 10% of dimensions suffices for convergence and stability across various models and datasets.

Findings

Applying RoPE to 10% of dimensions achieves similar convergence to full RoPE.

Partial RoPE significantly reduces memory usage, up to 10x savings.

Minimal RoPE application improves training stability, especially with NoPE.

Abstract

Rotary Positional Embedding (RoPE) is a common choice in transformer architectures for encoding relative positional information. Although earlier work has examined omitting RoPE in specific layers, the effect of varying the fraction of hidden dimensions that receive rotary transformations remains largely unexplored. This design choice can yield substantial memory savings, which becomes especially significant at long context lengths. We find up to 10x memory savings over the standard RoPE cache, while achieving comparable final loss. In this work, we present a systematic study examining the impact of partial RoPE on training dynamics and convergence across architectures and datasets. Our findings uncover several notable patterns: (1) applying RoPE to only a small fraction of dimensions (around 10%) achieves convergence comparable to using full RoPE; (2) these trends hold consistently across model size, sequence lengths and datasets of varying quality and architectures, with higher-quality data resulting in lower overall loss and similar benchmark performance; and (3) some models trained with NoPE (No Positional Encoding) showcase unstable learning trajectories, which can be alleviated through minimal RoPE application or QK-Norm which converges to a higher loss. Together, these results offer practical guidance for model designers aiming to balance efficiency and training stability, while emphasizing the previously overlooked importance of partial RoPE.