Rethinking Local Perception in Lightweight Vision Transformer

TL;DR

CloFormer is a lightweight vision transformer that effectively captures local and global features by combining context-aware local enhancement with attention mechanisms, improving performance across vision tasks.

Contribution

The paper introduces AttnConv, a novel convolution operator in attention style, and demonstrates how combining local and global information enhances lightweight vision transformers.

Findings

CloFormer outperforms existing lightweight models in image classification.

It achieves superior results in object detection and semantic segmentation.

The model reduces FLOPs while maintaining high accuracy.

Abstract

Vision Transformers (ViTs) have been shown to be effective in various vision tasks. However, resizing them to a mobile-friendly size leads to significant performance degradation. Therefore, developing lightweight vision transformers has become a crucial area of research. This paper introduces CloFormer, a lightweight vision transformer that leverages context-aware local enhancement. CloFormer explores the relationship between globally shared weights often used in vanilla convolutional operators and token-specific context-aware weights appearing in attention, then proposes an effective and straightforward module to capture high-frequency local information. In CloFormer, we introduce AttnConv, a convolution operator in attention's style. The proposed AttnConv uses shared weights to aggregate local information and deploys carefully designed context-aware weights to enhance local features. The combination of the AttnConv and vanilla attention which uses pooling to reduce FLOPs in CloFormer enables the model to perceive high-frequency and low-frequency information. Extensive experiments were conducted in image classification, object detection, and semantic segmentation, demonstrating the superiority of CloFormer. The code is available at \url{https://github.com/qhfan/CloFormer}.