Round and Round We Go! What makes Rotary Positional Encodings useful?

TL;DR

This paper investigates the internal mechanics of Rotary Positional Encodings in LLMs, revealing their use in constructing attention patterns and carrying semantic info, and proposes modifications to improve their effectiveness.

Contribution

It provides a detailed analysis of RoPE's role in LLMs, challenges common assumptions, and introduces a modified RoPE to enhance performance.

Findings

Gemma 7B learns to use RoPE for robust attention patterns.

Gemma prefers low frequencies of RoPE for semantic information.

Proposed RoPE modification improves model performance.

Abstract

Positional Encodings (PEs) are a critical component of Transformer-based Large Language Models (LLMs), providing the attention mechanism with important sequence-position information. One of the most popular types of encoding used today in LLMs are Rotary Positional Encodings (RoPE), that rotate the queries and keys based on their relative distance. A common belief is that RoPE is useful because it helps to decay token dependency as relative distance increases. In this work, we argue that this is unlikely to be the core reason. We study the internals of a trained Gemma 7B model to understand how RoPE is being used at a mechanical level. We find that Gemma learns to use RoPE to construct robust "positional" attention patterns by exploiting the highest frequencies. We also find that, in general, Gemma greatly prefers to use the lowest frequencies of RoPE, which we suspect are used to carry semantic information. We mathematically prove interesting behaviours of RoPE and conduct experiments to verify our findings, proposing a modification of RoPE that fixes some highlighted issues and improves performance. We believe that this work represents an interesting step in better understanding PEs in LLMs, which we believe holds crucial value for scaling LLMs to large sizes and context lengths.