Are Transformers More Robust Than CNNs?

TL;DR

This paper provides a fair comparison between Transformers and CNNs in visual recognition, revealing that CNNs can be as robust as Transformers with proper training, challenging previous beliefs about their robustness advantages.

Contribution

It offers the first fair, in-depth robustness comparison between Transformers and CNNs, showing CNNs can match Transformers' robustness with appropriate training methods.

Findings

CNNs can be as robust as Transformers with proper training

Pre-training on large datasets is not essential for Transformers' performance

Transformers' self-attention architecture benefits generalization

Abstract

Transformer emerges as a powerful tool for visual recognition. In addition to demonstrating competitive performance on a broad range of visual benchmarks, recent works also argue that Transformers are much more robust than Convolutions Neural Networks (CNNs). Nonetheless, surprisingly, we find these conclusions are drawn from unfair experimental settings, where Transformers and CNNs are compared at different scales and are applied with distinct training frameworks. In this paper, we aim to provide the first fair & in-depth comparisons between Transformers and CNNs, focusing on robustness evaluations. With our unified training setup, we first challenge the previous belief that Transformers outshine CNNs when measuring adversarial robustness. More surprisingly, we find CNNs can easily be as robust as Transformers on defending against adversarial attacks, if they properly adopt Transformers' training recipes. While regarding generalization on out-of-distribution samples, we show pre-training on (external) large-scale datasets is not a fundamental request for enabling Transformers to achieve better performance than CNNs. Moreover, our ablations suggest such stronger generalization is largely benefited by the Transformer's self-attention-like architectures per se, rather than by other training setups. We hope this work can help the community better understand and benchmark the robustness of Transformers and CNNs. The code and models are publicly available at https://github.com/ytongbai/ViTs-vs-CNNs.