An Impartial Take to the CNN vs Transformer Robustness Contest

TL;DR

This paper empirically compares CNNs and Transformers in terms of robustness and reliability, showing that recent CNNs like ConvNeXt can match or surpass Transformers, with no clear overall winner.

Contribution

It provides a comprehensive empirical analysis demonstrating that modern CNNs are as robust and reliable as Transformers, challenging the common belief of Transformers' superiority.

Findings

Recent CNNs can be as robust as Transformers

No architecture consistently outperforms the other

Both architectures share similar vulnerabilities

Abstract

Following the surge of popularity of Transformers in Computer Vision, several studies have attempted to determine whether they could be more robust to distribution shifts and provide better uncertainty estimates than Convolutional Neural Networks (CNNs). The almost unanimous conclusion is that they are, and it is often conjectured more or less explicitly that the reason of this supposed superiority is to be attributed to the self-attention mechanism. In this paper we perform extensive empirical analyses showing that recent state-of-the-art CNNs (particularly, ConvNeXt) can be as robust and reliable or even sometimes more than the current state-of-the-art Transformers. However, there is no clear winner. Therefore, although it is tempting to state the definitive superiority of one family of architectures over another, they seem to enjoy similar extraordinary performances on a variety of tasks while also suffering from similar vulnerabilities such as texture, background, and simplicity biases.