RegionViT: Regional-to-Local Attention for Vision Transformers

TL;DR

RegionViT introduces a regional-to-local attention mechanism within a pyramid structure for vision transformers, effectively capturing both global and local information, leading to improved performance across multiple vision tasks.

Contribution

The paper proposes a novel regional-to-local attention mechanism in a pyramid vision transformer architecture, enhancing global and local feature integration for better vision task performance.

Findings

Outperforms or matches state-of-the-art ViT variants on multiple tasks

Effective regional-to-local attention captures global and local information

Demonstrates versatility across classification, detection, segmentation, and recognition

Abstract

Vision transformer (ViT) has recently shown its strong capability in achieving comparable results to convolutional neural networks (CNNs) on image classification. However, vanilla ViT simply inherits the same architecture from the natural language processing directly, which is often not optimized for vision applications. Motivated by this, in this paper, we propose a new architecture that adopts the pyramid structure and employ a novel regional-to-local attention rather than global self-attention in vision transformers. More specifically, our model first generates regional tokens and local tokens from an image with different patch sizes, where each regional token is associated with a set of local tokens based on the spatial location. The regional-to-local attention includes two steps: first, the regional self-attention extract global information among all regional tokens and then the local self-attention exchanges the information among one regional token and the associated local tokens via self-attention. Therefore, even though local self-attention confines the scope in a local region but it can still receive global information. Extensive experiments on four vision tasks, including image classification, object and keypoint detection, semantics segmentation and action recognition, show that our approach outperforms or is on par with state-of-the-art ViT variants including many concurrent works. Our source codes and models are available at https://github.com/ibm/regionvit.