Global Context Vision Transformers

TL;DR

The paper introduces GC ViT, a global context vision transformer that efficiently models long and short-range interactions, achieving state-of-the-art results in image classification, detection, and segmentation without expensive operations.

Contribution

It presents a novel architecture combining global and local self-attention, along with modified residual blocks, to improve parameter and compute efficiency in vision transformers.

Findings

GC ViT achieves 85.7% Top-1 accuracy on ImageNet-1K without pre-training.

Pre-trained GC ViT outperforms prior models in object detection and segmentation tasks.

GC ViT surpasses CNN-based and other ViT-based models in multiple vision benchmarks.

Abstract

We propose global context vision transformer (GC ViT), a novel architecture that enhances parameter and compute utilization for computer vision. Our method leverages global context self-attention modules, joint with standard local self-attention, to effectively and efficiently model both long and short-range spatial interactions, without the need for expensive operations such as computing attention masks or shifting local windows. In addition, we address the lack of the inductive bias in ViTs, and propose to leverage a modified fused inverted residual blocks in our architecture. Our proposed GC ViT achieves state-of-the-art results across image classification, object detection and semantic segmentation tasks. On ImageNet-1K dataset for classification, the variants of GC ViT with 51M, 90M and 201M parameters achieve 84.3%, 85.0% and 85.7% Top-1 accuracy, respectively, at 224 image resolution and without any pre-training, hence surpassing comparably-sized prior art such as CNN-based ConvNeXt and ViT-based MaxViT and Swin Transformer by a large margin. Pre-trained GC ViT backbones in downstream tasks of object detection, instance segmentation, and semantic segmentation using MS COCO and ADE20K datasets outperform prior work consistently. Specifically, GC ViT with a 4-scale DINO detection head achieves a box AP of 58.3 on MS COCO dataset.