On the Robustness of Vision Transformers to Adversarial Examples

TL;DR

This paper investigates the robustness of Vision Transformers against adversarial attacks, revealing their transferability properties, vulnerabilities, and potential for ensemble defenses, with extensive experiments on multiple datasets and attack types.

Contribution

It provides the first comprehensive analysis of Vision Transformers' adversarial robustness, including new attack methods and ensemble defense strategies.

Findings

Adversarial examples do not transfer easily between CNNs and transformers.

Ensemble of CNNs and transformers can be robust under black-box attacks.

Transformers are vulnerable to white-box attacks despite high accuracy.

Abstract

Recent advances in attention-based networks have shown that Vision Transformers can achieve state-of-the-art or near state-of-the-art results on many image classification tasks. This puts transformers in the unique position of being a promising alternative to traditional convolutional neural networks (CNNs). While CNNs have been carefully studied with respect to adversarial attacks, the same cannot be said of Vision Transformers. In this paper, we study the robustness of Vision Transformers to adversarial examples. Our analyses of transformer security is divided into three parts. First, we test the transformer under standard white-box and black-box attacks. Second, we study the transferability of adversarial examples between CNNs and transformers. We show that adversarial examples do not readily transfer between CNNs and transformers. Based on this finding, we analyze the security of a simple ensemble defense of CNNs and transformers. By creating a new attack, the self-attention blended gradient attack, we show that such an ensemble is not secure under a white-box adversary. However, under a black-box adversary, we show that an ensemble can achieve unprecedented robustness without sacrificing clean accuracy. Our analysis for this work is done using six types of white-box attacks and two types of black-box attacks. Our study encompasses multiple Vision Transformers, Big Transfer Models and CNN architectures trained on CIFAR-10, CIFAR-100 and ImageNet.