S2AFormer: Strip Self-Attention for Efficient Vision Transformer

TL;DR

S2AFormer introduces a novel Strip Self-Attention mechanism that reduces computational complexity in Vision Transformers, combining CNN local perception with global attention, leading to efficient and accurate vision models.

Contribution

The paper proposes S2AFormer with Strip Self-Attention, a new method that significantly reduces computation while maintaining accuracy in Vision Transformers.

Findings

Achieves higher accuracy on ImageNet-1k with less computation.

Demonstrates superior efficiency on multiple vision benchmarks.

Maintains robustness across different hardware environments.

Abstract

Vision Transformer (ViT) has made significant advancements in computer vision, thanks to its token mixer's sophisticated ability to capture global dependencies between all tokens. However, the quadratic growth in computational demands as the number of tokens increases limits its practical efficiency. Although recent methods have combined the strengths of convolutions and self-attention to achieve better trade-offs, the expensive pairwise token affinity and complex matrix operations inherent in self-attention remain a bottleneck. To address this challenge, we propose S2AFormer, an efficient Vision Transformer architecture featuring novel Strip Self-Attention (SSA). We design simple yet effective Hybrid Perception Blocks (HPBs) to effectively integrate the local perception capabilities of CNNs with the global context modeling of Transformer's attention mechanisms. A key innovation of SSA lies in its reduction of the spatial dimensions of $K$ and $V$, while compressing the channel dimensions of $Q$ and $K$. This design significantly reduces computational overhead while preserving accuracy, striking an optimal balance between efficiency and effectiveness. We evaluate the robustness and efficiency of S2AFormer through extensive experiments on multiple vision benchmarks, including ImageNet-1k for image classification, ADE20k for semantic segmentation, and COCO for object detection and instance segmentation. Results demonstrate that S2AFormer achieves significant accuracy gains with superior efficiency in both GPU and non-GPU environments, making it a strong candidate for efficient vision Transformers.