Vision Transformers are Circulant Attention Learners

TL;DR

This paper introduces Circulant Attention, a novel efficient attention mechanism for vision Transformers that exploits the inherent block circulant structure of self-attention matrices to reduce computational complexity from quadratic to near-linear.

Contribution

The paper proposes modeling self-attention as a block circulant matrix, enabling fast computation with $ ext{O}(N ext{log}N)$ complexity while maintaining model capacity.

Findings

Achieves $ ext{O}(N ext{log}N)$ complexity in attention computation.

Maintains comparable performance to vanilla self-attention on visual tasks.

Provides a practical alternative for high-resolution vision Transformer models.

Abstract

The self-attention mechanism has been a key factor in the advancement of vision Transformers. However, its quadratic complexity imposes a heavy computational burden in high-resolution scenarios, restricting the practical application. Previous methods attempt to mitigate this issue by introducing handcrafted patterns such as locality or sparsity, which inevitably compromise model capacity. In this paper, we present a novel attention paradigm termed \textbf{Circulant Attention} by exploiting the inherent efficient pattern of self-attention. Specifically, we first identify that the self-attention matrix in vision Transformers often approximates the Block Circulant matrix with Circulant Blocks (BCCB), a kind of structured matrix whose multiplication with other matrices can be performed in $\mathcal{O}(N\log N)$ time. Leveraging this interesting pattern, we explicitly model the attention map as its nearest BCCB matrix and propose an efficient computation algorithm for fast calculation. The resulting approach closely mirrors vanilla self-attention, differing only in its use of BCCB matrices. Since our design is inspired by the inherent efficient paradigm, it not only delivers $\mathcal{O}(N\log N)$ computation complexity, but also largely maintains the capacity of standard self-attention. Extensive experiments on diverse visual tasks demonstrate the effectiveness of our approach, establishing circulant attention as a promising alternative to self-attention for vision Transformer architectures. Code is available at https://github.com/LeapLabTHU/Circulant-Attention.