SimViT: Exploring a Simple Vision Transformer with sliding windows

TL;DR

SimViT is a simple vision Transformer that incorporates local spatial structure using sliding windows and a novel attention mechanism, achieving high accuracy with fewer parameters for image recognition tasks.

Contribution

Introduces SimViT, a vision Transformer with Multi-head Central Self-Attention and sliding windows to better capture local relations and spatial structure.

Findings

SimViT-Micro achieves 71.1% top-1 accuracy on ImageNet-1k with only 3.3M parameters.

SimViT outperforms some existing models in efficiency and effectiveness.

The model is suitable as a general-purpose backbone for various vision tasks.

Abstract

Although vision Transformers have achieved excellent performance as backbone models in many vision tasks, most of them intend to capture global relations of all tokens in an image or a window, which disrupts the inherent spatial and local correlations between patches in 2D structure. In this paper, we introduce a simple vision Transformer named SimViT, to incorporate spatial structure and local information into the vision Transformers. Specifically, we introduce Multi-head Central Self-Attention(MCSA) instead of conventional Multi-head Self-Attention to capture highly local relations. The introduction of sliding windows facilitates the capture of spatial structure. Meanwhile, SimViT extracts multi-scale hierarchical features from different layers for dense prediction tasks. Extensive experiments show the SimViT is effective and efficient as a general-purpose backbone model for various image processing tasks. Especially, our SimViT-Micro only needs 3.3M parameters to achieve 71.1% top-1 accuracy on ImageNet-1k dataset, which is the smallest size vision Transformer model by now. Our code will be available in https://github.com/ucasligang/SimViT.