A Length-Extrapolatable Transformer

TL;DR

This paper introduces a novel Transformer design that enhances length extrapolation capabilities by improving attention resolution through relative position embeddings and blockwise causal attention, enabling better performance on longer sequences.

Contribution

The paper proposes two new methods to improve length extrapolation in Transformers, focusing on attention resolution with relative position embeddings and blockwise causal attention.

Findings

Enhanced performance in length extrapolation tasks

Improved attention resolution metrics

Effective in language modeling for longer sequences

Abstract

Position modeling plays a critical role in Transformers. In this paper, we focus on length extrapolation, i.e., training on short texts while evaluating longer sequences. We define attention resolution as an indicator of extrapolation. Then we propose two designs to improve the above metric of Transformers. Specifically, we introduce a relative position embedding to explicitly maximize attention resolution. Moreover, we use blockwise causal attention during inference for better resolution. We evaluate different Transformer variants with language modeling. Experimental results show that our model achieves strong performance in both interpolation and extrapolation settings. The code will be available at https://aka.ms/LeX-Transformer.