Linearized Relative Positional Encoding

TL;DR

This paper introduces a new family of linear relative positional encoding algorithms, LRPE, that are compatible with linear transformers and achieve state-of-the-art results across multiple tasks.

Contribution

It develops a principled framework for designing linear relative positional encodings using unitary transformations, enabling effective and efficient encoding for linear transformers.

Findings

LRPE achieves state-of-the-art performance in language modeling.

LRPE outperforms existing methods in text and image classification.

The framework generalizes to various applications and preserves linear complexity.

Abstract

Relative positional encoding is widely used in vanilla and linear transformers to represent positional information. However, existing encoding methods of a vanilla transformer are not always directly applicable to a linear transformer, because the latter requires a decomposition of the query and key representations into separate kernel functions. Nevertheless, principles for designing encoding methods suitable for linear transformers remain understudied. In this work, we put together a variety of existing linear relative positional encoding approaches under a canonical form and further propose a family of linear relative positional encoding algorithms via unitary transformation. Our formulation leads to a principled framework that can be used to develop new relative positional encoding methods that preserve linear space-time complexity. Equipped with different models, the proposed linearized relative positional encoding (LRPE) family derives effective encoding for various applications. Experiments show that compared with existing methods, LRPE achieves state-of-the-art performance in language modeling, text classification, and image classification. Meanwhile, it emphasizes a general paradigm for designing broadly more relative positional encoding methods that are applicable to linear transformers. The code is available at https://github.com/OpenNLPLab/Lrpe.