Fibottention: Inceptive Visual Representation Learning with Diverse Attention Across Heads

TL;DR

Fibottention introduces a structured sparse self-attention mechanism based on the Wythoff array, reducing computational complexity and enhancing diversity in vision transformers, leading to improved efficiency and performance across multiple visual tasks.

Contribution

The paper proposes Fibottention, a novel sparse self-attention method with diverse head patterns, achieving lower complexity and better feature diversity in vision transformers.

Findings

Achieves $ ext{O}(N ext{log} N)$ complexity in self-attention.

Models with Fibottention outperform or match dense models with only 2% of pairwise interactions.

Consistently superior results compared to existing sparse attention methods.

Abstract

Vision Transformers and their variants have achieved remarkable success in diverse visual perception tasks. Despite their effectiveness, they suffer from two significant limitations. First, the quadratic computational complexity of multi-head self-attention (MHSA), which restricts scalability to large token counts, and second, a high dependency on large-scale training data to attain competitive performance. In this paper, to address these challenges, we propose a novel sparse self-attention mechanism named Fibottention. Fibottention employs structured sparsity patterns derived from the Wythoff array, enabling an $\mathcal{O}(N \log N)$ computational complexity in self-attention. By design, its sparsity patterns vary across attention heads, which provably reduces redundant pairwise interactions while ensuring sufficient and diverse coverage. This leads to an \emph{inception-like functional diversity} in the attention heads, and promotes more informative and disentangled representations. We integrate Fibottention into standard Transformer architectures and conduct extensive experiments across multiple domains, including image classification, video understanding, and robot learning. Results demonstrate that models equipped with Fibottention either significantly outperform or achieve on-par performance with their dense MHSA counterparts, while leveraging only $2\%$ of all pairwise interactions across self-attention heads in typical settings, $2-6\%$ of the pairwise interactions in self-attention heads, resulting in substantial computational savings. Moreover, when compared to existing sparse attention mechanisms, Fibottention consistently achieves superior results on a FLOP-equivalency basis. Finally, we provide an in-depth analysis of the enhanced feature diversity resulting from our attention design and discuss its implications for efficient representation learning.