Context-aware Rotary Position Embedding

TL;DR

CARoPE introduces a dynamic, context-sensitive rotary positional embedding for Transformers, improving modeling of sequence relationships, reducing perplexity, and increasing training efficiency over static RoPE.

Contribution

This work presents CARoPE, a novel positional encoding that adapts rotary embeddings based on token context, enhancing expressiveness and efficiency in Transformer models.

Findings

CARoPE outperforms RoPE in perplexity on the FineWeb-Edu-10B dataset.

CARoPE enables faster training throughput without losing stability.

It achieves better modeling of sequence relationships at longer context lengths.

Abstract

Positional encoding is a vital component of Transformer architectures, enabling models to incorporate sequence order into self-attention mechanisms. Rotary Positional Embeddings (RoPE) have become a widely adopted solution due to their compatibility with relative position encoding and computational efficiency. However, RoPE relies on static, input-independent sinusoidal frequency patterns, limiting its ability to model context-sensitive relationships. In this work, we propose CARoPE (Context-Aware Rotary Positional Embedding), a novel generalization of RoPE that dynamically generates head-specific frequency patterns conditioned on token embeddings. This design introduces token- and context-sensitive positional representations while preserving RoPE efficiency and architectural simplicity. CARoPE computes input-dependent phase shifts using a bounded transformation of token embeddings and integrates them into the rotary mechanism across attention heads. We evaluate CARoPE on the FineWeb-Edu-10B dataset using GPT-2 variants trained on next-token prediction tasks. Experimental results show that CARoPE consistently outperforms RoPE and other common positional encoding baselines, achieving significantly lower perplexity, even at longer context lengths. Additionally, CARoPE enables faster training throughput without sacrificing model stability. These findings demonstrate that CARoPE offers a scalable, expressive, and efficient upgrade to existing positional encoding strategies in Transformer models.