3D-RPE: Enhancing Long-Context Modeling Through 3D Rotary Position Encoding

TL;DR

This paper introduces 3D Rotary Position Encoding (3D-RPE), a novel method inspired by the Bloch Sphere, that improves long-context modeling by controlling decay and enhancing position resolution, leading to better performance in long-sequence NLP tasks.

Contribution

The paper proposes 3D-RPE, an advanced rotary position encoding method that offers controllable decay and improved resolution, outperforming traditional RoPE in long-context NLP tasks.

Findings

3D-RPE outperforms RoPE in long-context NLU tasks.

3D-RPE provides controllable long-term decay for better relative position modeling.

3D-RPE mitigates position resolution degradation caused by interpolation.

Abstract

Inspired by the Bloch Sphere representation, we propose a novel rotary position encoding on a three-dimensional sphere, named 3D Rotary Position Encoding (3D-RPE). 3D-RPE is an advanced version of the widely used 2D Rotary Position Encoding (RoPE), with two major advantages for modeling long contexts: controllable long-term decay and improved position resolution. For controllable long-term decay, 3D-RPE allows for the regulation of long-term decay within the chunk size, ensuring the modeling of relative positional information between tokens at a distant relative position. For enhanced position resolution, 3D-RPE can mitigate the degradation of position resolution caused by position interpolation on RoPE. We have conducted experiments on long-context Natural Language Understanding (NLU) and long-sequence Language Modeling (LM) tasks. From the experimental results, 3D-RPE achieved performance improvements over RoPE, especially in long-context NLU tasks.