PSC: Extending Context Window of Large Language Models via Phase Shift Calibration

TL;DR

This paper introduces PSC, a calibration module that enhances existing position encoding methods like RoPE, enabling large language models to effectively extend their context windows up to 64k tokens with improved performance.

Contribution

PSC provides a simple calibration approach that improves the effectiveness of existing frequency rescaling methods for expanding LLM context windows.

Findings

PSC improves perplexity reductions across larger context windows

The method is broadly applicable across models and tasks

Enhances existing position encoding techniques like PI, YaRN, and LongRoPE

Abstract

Rotary Position Embedding (RoPE) is an efficient position encoding approach and is widely utilized in numerous large language models (LLMs). Recently, a lot of methods have been put forward to further expand the context window based on RoPE. The core concept of those methods is to predefine or search for a set of factors to rescale the base frequencies of RoPE. Nevertheless, it is quite a challenge for existing methods to predefine an optimal factor due to the exponential search space. In view of this, we introduce PSC (Phase Shift Calibration), a small module for calibrating the frequencies predefined by existing methods. With the employment of PSC, we demonstrate that many existing methods can be further enhanced, like PI, YaRN, and LongRoPE. We conducted extensive experiments across multiple models and tasks. The results demonstrate that (1) when PSC is enabled, the comparative reductions in perplexity increase as the context window size is varied from 16k, to 32k, and up to 64k. (2) Our approach is broadly applicable and exhibits robustness across a variety of models and tasks. The code can be found at https://github.com/WNQzhu/PSC.