Are Vision Transformers Robust to Spurious Correlations?

TL;DR

This paper investigates how vision transformers handle spurious correlations, finding that pre-trained ViT models are more robust than CNNs, especially when generalizing from non-spurious examples, with the self-attention mechanism playing a key role.

Contribution

It provides a systematic comparison of ViT and CNN robustness to spurious correlations and explores the role of self-attention in enhancing robustness.

Findings

Pre-trained ViT models outperform CNNs in robustness.

ViTs generalize better from non-spurious examples.

Self-attention contributes to robustness.

Abstract

Deep neural networks may be susceptible to learning spurious correlations that hold on average but not in atypical test samples. As with the recent emergence of vision transformer (ViT) models, it remains underexplored how spurious correlations are manifested in such architectures. In this paper, we systematically investigate the robustness of vision transformers to spurious correlations on three challenging benchmark datasets and compare their performance with popular CNNs. Our study reveals that when pre-trained on a sufficiently large dataset, ViT models are more robust to spurious correlations than CNNs. Key to their success is the ability to generalize better from the examples where spurious correlations do not hold. Further, we perform extensive ablations and experiments to understand the role of the self-attention mechanism in providing robustness under spuriously correlated environments. We hope that our work will inspire future research on further understanding the robustness of ViT models.