Is Attention Better Than Matrix Decomposition?

TL;DR

This paper challenges the supremacy of self-attention in deep learning by showing that matrix decomposition methods, modeled as low-rank recovery problems, can perform comparably or better in encoding global context with less computational cost.

Contribution

It introduces Hamburgers, a novel approach using matrix decomposition algorithms to model global context, providing an alternative to self-attention with demonstrated effectiveness in vision tasks.

Findings

Matrix decomposition can match or outperform self-attention in global context modeling.

Hamburgers improve performance in semantic segmentation and image generation.

The approach reduces computational costs compared to traditional self-attention.

Abstract

As an essential ingredient of modern deep learning, attention mechanism, especially self-attention, plays a vital role in the global correlation discovery. However, is hand-crafted attention irreplaceable when modeling the global context? Our intriguing finding is that self-attention is not better than the matrix decomposition (MD) model developed 20 years ago regarding the performance and computational cost for encoding the long-distance dependencies. We model the global context issue as a low-rank recovery problem and show that its optimization algorithms can help design global information blocks. This paper then proposes a series of Hamburgers, in which we employ the optimization algorithms for solving MDs to factorize the input representations into sub-matrices and reconstruct a low-rank embedding. Hamburgers with different MDs can perform favorably against the popular global context module self-attention when carefully coping with gradients back-propagated through MDs. Comprehensive experiments are conducted in the vision tasks where it is crucial to learn the global context, including semantic segmentation and image generation, demonstrating significant improvements over self-attention and its variants.