Extending the Context of Pretrained LLMs by Dropping Their Positional Embeddings

TL;DR

This paper introduces DroPE, a simple method that removes positional embeddings after pretraining, enabling language models to extend their context seamlessly without additional finetuning.

Contribution

The paper demonstrates that positional embeddings can be dropped post-pretraining, allowing models to generalize to longer sequences without finetuning, challenging prior assumptions.

Findings

DroPE enables zero-shot extension of context length.

Models maintain performance without long-context finetuning.

Outperforms previous methods in extending context length.

Abstract

So far, expensive finetuning beyond the pretraining sequence length has been a requirement for effectively extending the context of language models (LM). In this work, we break this key bottleneck by Dropping the Positional Embeddings of LMs after training (DroPE). Our simple method is motivated by three key theoretical and empirical observations. First, positional embeddings (PEs) serve a crucial role during pretraining, providing an important inductive bias that significantly facilitates convergence. Second, over-reliance on this explicit positional information is also precisely what prevents test-time generalization to sequences of unseen length, even when using popular PE-scaling methods. Third, positional embeddings are not an inherent requirement of effective language modeling and can be safely removed after pretraining, following a short recalibration phase. Empirically, DroPE yields seamless zero-shot context extension without any long-context finetuning, quickly adapting pretrained LMs without compromising their capabilities in the original training context. Our findings hold across different models and dataset sizes, far outperforming previous specialized architectures and established rotary positional embedding scaling methods.