Base of RoPE Bounds Context Length

TL;DR

This paper investigates the role of Rotary Position Embedding (RoPE) in LLMs, revealing a theoretical lower bound on the RoPE base parameter necessary for achieving certain context lengths, impacting long-context training.

Contribution

It introduces the concept of the RoPE base bounds for context length, providing both theoretical analysis and empirical evidence of its importance.

Findings

There is a lower bound on the RoPE base for desired context length.

Superficial long-context ability may be due to out-of-distribution effects.

The relationship between RoPE base and context length is both theoretically and empirically established.

Abstract

Position embedding is a core component of current Large Language Models (LLMs). Rotary position embedding (RoPE), a technique that encodes the position information with a rotation matrix, has been the de facto choice for position embedding in many LLMs, such as the Llama series. RoPE has been further utilized to extend long context capability, which is roughly based on adjusting the \textit{base} parameter of RoPE to mitigate out-of-distribution (OOD) problems in position embedding. However, in this paper, we find that LLMs may obtain a superficial long-context ability based on the OOD theory. We revisit the role of RoPE in LLMs and propose a novel property of long-term decay, we derive that the \textit{base of RoPE bounds context length}: there is an absolute lower bound for the base value to obtain certain context length capability. Our work reveals the relationship between context length and RoPE base both theoretically and empirically, which may shed light on future long context training.