Linformer: Self-Attention with Linear Complexity

TL;DR

The paper introduces Linformer, a linear-time self-attention mechanism for transformers that maintains performance while significantly reducing computational and memory costs for long sequences.

Contribution

It proposes a low-rank approximation of self-attention, enabling linear complexity in sequence length, which is a novel approach to improving transformer efficiency.

Findings

Linformer matches standard transformer performance.

Reduces self-attention complexity from O(n^2) to O(n).

Demonstrates efficiency gains on long sequences.

Abstract

Large transformer models have shown extraordinary success in achieving state-of-the-art results in many natural language processing applications. However, training and deploying these models can be prohibitively costly for long sequences, as the standard self-attention mechanism of the Transformer uses $O(n^2)$ time and space with respect to sequence length. In this paper, we demonstrate that the self-attention mechanism can be approximated by a low-rank matrix. We further exploit this finding to propose a new self-attention mechanism, which reduces the overall self-attention complexity from $O(n^2)$ to $O(n)$ in both time and space. The resulting linear transformer, the \textit{Linformer}, performs on par with standard Transformer models, while being much more memory- and time-efficient.