Query-Efficient Hard-Label Black-Box Attack against Vision Transformers

TL;DR

This paper introduces AdvViT, a query-efficient hard-label black-box attack tailored for Vision Transformers, exploiting their patch sensitivity to demonstrate their vulnerability with minimal perturbations.

Contribution

The paper presents a novel attack method specifically designed for ViTs, optimizing perturbations on patches and low-frequency components to improve attack efficiency and effectiveness.

Findings

AdvViT achieves lower $L_2$-norm distortion than CNN attacks.

The attack is highly query-efficient on multiple ViT models.

ViTs are vulnerable to adversarial attacks even in black-box scenarios.

Abstract

Recent studies have revealed that vision transformers (ViTs) face similar security risks from adversarial attacks as deep convolutional neural networks (CNNs). However, directly applying attack methodology on CNNs to ViTs has been demonstrated to be ineffective since the ViTs typically work on patch-wise encoding. This article explores the vulnerability of ViTs against adversarial attacks under a black-box scenario, and proposes a novel query-efficient hard-label adversarial attack method called AdvViT. Specifically, considering that ViTs are highly sensitive to patch modification, we propose to optimize the adversarial perturbation on the individual patches. To reduce the dimension of perturbation search space, we modify only a handful of low-frequency components of each patch. Moreover, we design a weight mask matrix for all patches to further optimize the perturbation on different regions of a whole image. We test six mainstream ViT backbones on the ImageNet-1k dataset. Experimental results show that compared with the state-of-the-art attacks on CNNs, our AdvViT achieves much lower $L_2$-norm distortion under the same query budget, sufficiently validating the vulnerability of ViTs against adversarial attacks.