Intriguing Properties of Vision Transformers

TL;DR

This paper explores the robust and shape-aware properties of Vision Transformers (ViT), demonstrating their resilience to occlusions and perturbations, their shape bias, and their effectiveness in segmentation and ensemble learning.

Contribution

The study provides a comprehensive analysis of ViT's properties, revealing their robustness, shape bias, and potential for diverse vision tasks, which were previously underexplored.

Findings

ViTs retain high accuracy under severe occlusions and domain shifts.

ViTs are less biased towards textures and can recognize shapes similarly to humans.

Single ViT features can be combined for high-accuracy classification and few-shot learning.

Abstract

Vision transformers (ViT) have demonstrated impressive performance across various machine vision problems. These models are based on multi-head self-attention mechanisms that can flexibly attend to a sequence of image patches to encode contextual cues. An important question is how such flexibility in attending image-wide context conditioned on a given patch can facilitate handling nuisances in natural images e.g., severe occlusions, domain shifts, spatial permutations, adversarial and natural perturbations. We systematically study this question via an extensive set of experiments encompassing three ViT families and comparisons with a high-performing convolutional neural network (CNN). We show and analyze the following intriguing properties of ViT: (a) Transformers are highly robust to severe occlusions, perturbations and domain shifts, e.g., retain as high as 60% top-1 accuracy on ImageNet even after randomly occluding 80% of the image content. (b) The robust performance to occlusions is not due to a bias towards local textures, and ViTs are significantly less biased towards textures compared to CNNs. When properly trained to encode shape-based features, ViTs demonstrate shape recognition capability comparable to that of human visual system, previously unmatched in the literature. (c) Using ViTs to encode shape representation leads to an interesting consequence of accurate semantic segmentation without pixel-level supervision. (d) Off-the-shelf features from a single ViT model can be combined to create a feature ensemble, leading to high accuracy rates across a range of classification datasets in both traditional and few-shot learning paradigms. We show effective features of ViTs are due to flexible and dynamic receptive fields possible via the self-attention mechanism.