Visual Transformers: Token-based Image Representation and Processing for Computer Vision

TL;DR

This paper introduces Visual Transformers that operate on semantic tokens instead of pixels, enabling more efficient and accurate image understanding by modeling relationships across image parts with less computation.

Contribution

It proposes a novel semantic token-based transformer approach for computer vision, outperforming convolutional models in accuracy and efficiency.

Findings

ResNet accuracy on ImageNet improved by 4.6 to 7 points.

FPN-based segmentation achieves 0.35 higher mIoU.

FPN FLOPs reduced by 6.5x.

Abstract

Computer vision has achieved remarkable success by (a) representing images as uniformly-arranged pixel arrays and (b) convolving highly-localized features. However, convolutions treat all image pixels equally regardless of importance; explicitly model all concepts across all images, regardless of content; and struggle to relate spatially-distant concepts. In this work, we challenge this paradigm by (a) representing images as semantic visual tokens and (b) running transformers to densely model token relationships. Critically, our Visual Transformer operates in a semantic token space, judiciously attending to different image parts based on context. This is in sharp contrast to pixel-space transformers that require orders-of-magnitude more compute. Using an advanced training recipe, our VTs significantly outperform their convolutional counterparts, raising ResNet accuracy on ImageNet top-1 by 4.6 to 7 points while using fewer FLOPs and parameters. For semantic segmentation on LIP and COCO-stuff, VT-based feature pyramid networks (FPN) achieve 0.35 points higher mIoU while reducing the FPN module's FLOPs by 6.5x.