The Rotary Position Embedding May Cause Dimension Inefficiency in Attention Heads for Long-Distance Retrieval

TL;DR

This paper investigates how Rotary Position Embedding (RoPE) in large language models may lead to dimension inefficiency, especially in long-distance retrieval tasks, by causing some dimensions to be underutilized due to wide rotation angles.

Contribution

The study provides empirical evidence that RoPE can cause certain dimensions to be underused, highlighting a potential limitation of RoPE in long-context modeling.

Findings

RoPE causes low utility in some dimensions during attention.

Dimensions with large rotation angles are less effective.

RoPE's dimension inefficiency impacts long-distance question answering.

Abstract

The Rotary Position Embedding (RoPE) is widely used in the attention heads of many large language models (LLM). It rotates dimensions in the query and the key vectors by different angles according to their positions in the input sequence. For long context modeling, the range of positions may vary a lot, and thus RoPE rotates some dimensions by a great range of angles. We hypothesize that the wide range of rotation angles may prevent LLMs from utilizing those dimensions. To validate this hypothesis, we present a controlled experiment showing that applying RoPE causes low utility of certain dimensions. Our analyses on three LLMs also indicate that these dimensions do not help LLMs do long-context question answering.