Position Interpolation Improves ALiBi Extrapolation

TL;DR

This paper demonstrates that linear position interpolation enhances the ability of models with ALiBi to extrapolate to longer sequences, improving performance on language modeling, summarization, and retrieval tasks.

Contribution

It introduces the use of linear position interpolation to extend ALiBi's extrapolation range, a novel approach for better long-sequence modeling.

Findings

Position interpolation significantly improves extrapolation in language models.

Enhanced performance on downstream summarization and retrieval tasks.

Method is effective for models using Attention with Linear Biases (ALiBi).

Abstract

Linear position interpolation helps pre-trained models using rotary position embeddings (RoPE) to extrapolate to longer sequence lengths. We propose using linear position interpolation to extend the extrapolation range of models using Attention with Linear Biases (ALiBi). We find position interpolation significantly improves extrapolation capability on upstream language modelling and downstream summarization and retrieval tasks.