Bottleneck Transformers for Visual Recognition

TL;DR

BoTNet introduces a simple modification to ResNet by replacing some convolutions with self-attention, significantly improving performance in image recognition, detection, and segmentation tasks with minimal latency increase.

Contribution

The paper demonstrates that integrating self-attention into ResNet bottleneck blocks enhances vision tasks and provides a new perspective on viewing these blocks as Transformer components.

Findings

Achieves 44.4% Mask AP on COCO with Mask R-CNN.

Attains 84.7% top-1 accuracy on ImageNet.

Faster and more parameter-efficient than comparable models.

Abstract

We present BoTNet, a conceptually simple yet powerful backbone architecture that incorporates self-attention for multiple computer vision tasks including image classification, object detection and instance segmentation. By just replacing the spatial convolutions with global self-attention in the final three bottleneck blocks of a ResNet and no other changes, our approach improves upon the baselines significantly on instance segmentation and object detection while also reducing the parameters, with minimal overhead in latency. Through the design of BoTNet, we also point out how ResNet bottleneck blocks with self-attention can be viewed as Transformer blocks. Without any bells and whistles, BoTNet achieves 44.4% Mask AP and 49.7% Box AP on the COCO Instance Segmentation benchmark using the Mask R-CNN framework; surpassing the previous best published single model and single scale results of ResNeSt evaluated on the COCO validation set. Finally, we present a simple adaptation of the BoTNet design for image classification, resulting in models that achieve a strong performance of 84.7% top-1 accuracy on the ImageNet benchmark while being up to 1.64x faster in compute time than the popular EfficientNet models on TPU-v3 hardware. We hope our simple and effective approach will serve as a strong baseline for future research in self-attention models for vision