Efficient Attention: Attention with Linear Complexities

TL;DR

This paper introduces an efficient attention mechanism that reduces memory and computational costs from quadratic to linear, enabling its use in high-resolution inputs and improving performance in vision and language tasks.

Contribution

It proposes a novel attention method equivalent to dot-product attention but with linear complexity, facilitating broader application and better accuracy in complex models.

Findings

Significant performance improvements in object detection and segmentation on MS-COCO 2017.

Achieved state-of-the-art results in stereo depth estimation on Scene Flow dataset.

Democratizes attention mechanisms for resource-constrained models.

Abstract

Dot-product attention has wide applications in computer vision and natural language processing. However, its memory and computational costs grow quadratically with the input size. Such growth prohibits its application on high-resolution inputs. To remedy this drawback, this paper proposes a novel efficient attention mechanism equivalent to dot-product attention but with substantially less memory and computational costs. Its resource efficiency allows more widespread and flexible integration of attention modules into a network, which leads to better accuracies. Empirical evaluations demonstrated the effectiveness of its advantages. Efficient attention modules brought significant performance boosts to object detectors and instance segmenters on MS-COCO 2017. Further, the resource efficiency democratizes attention to complex models, where high costs prohibit the use of dot-product attention. As an exemplar, a model with efficient attention achieved state-of-the-art accuracies for stereo depth estimation on the Scene Flow dataset. Code is available at https://github.com/cmsflash/efficient-attention.