Reformer: The Efficient Transformer

TL;DR

The Reformer introduces locality-sensitive hashing attention and reversible residual layers to significantly improve the efficiency and memory usage of Transformer models, especially on long sequences, without sacrificing performance.

Contribution

It presents a novel combination of hashing-based attention and reversible layers to enhance Transformer efficiency and scalability.

Findings

Achieves comparable performance to standard Transformers.

Reduces memory usage during training.

Speeds up processing of long sequences.

Abstract

Large Transformer models routinely achieve state-of-the-art results on a number of tasks but training these models can be prohibitively costly, especially on long sequences. We introduce two techniques to improve the efficiency of Transformers. For one, we replace dot-product attention by one that uses locality-sensitive hashing, changing its complexity from O($L^2$) to O($L\log L$), where $L$ is the length of the sequence. Furthermore, we use reversible residual layers instead of the standard residuals, which allows storing activations only once in the training process instead of $N$ times, where $N$ is the number of layers. The resulting model, the Reformer, performs on par with Transformer models while being much more memory-efficient and much faster on long sequences.