Vision Transformers are Robust Learners

TL;DR

This paper evaluates the robustness of Vision Transformers (ViT) against various challenges and compares their performance with CNNs, revealing that ViTs are more robust learners due to unique properties like Fourier spectrum sensitivity.

Contribution

It provides a comprehensive robustness evaluation of ViT models, including novel analyses explaining their superior robustness over CNNs.

Findings

ViT achieves 4.3x higher accuracy on ImageNet-A compared to BiT.

ViT models show enhanced robustness against corruptions and adversarial examples.

Analyses reveal properties like Fourier spectrum sensitivity contribute to ViT robustness.

Abstract

Transformers, composed of multiple self-attention layers, hold strong promises toward a generic learning primitive applicable to different data modalities, including the recent breakthroughs in computer vision achieving state-of-the-art (SOTA) standard accuracy. What remains largely unexplored is their robustness evaluation and attribution. In this work, we study the robustness of the Vision Transformer (ViT) against common corruptions and perturbations, distribution shifts, and natural adversarial examples. We use six different diverse ImageNet datasets concerning robust classification to conduct a comprehensive performance comparison of ViT models and SOTA convolutional neural networks (CNNs), Big-Transfer. Through a series of six systematically designed experiments, we then present analyses that provide both quantitative and qualitative indications to explain why ViTs are indeed more robust learners. For example, with fewer parameters and similar dataset and pre-training combinations, ViT gives a top-1 accuracy of 28.10% on ImageNet-A which is 4.3x higher than a comparable variant of BiT. Our analyses on image masking, Fourier spectrum sensitivity, and spread on discrete cosine energy spectrum reveal intriguing properties of ViT attributing to improved robustness. Code for reproducing our experiments is available at https://git.io/J3VO0.