Transferable Adversarial Attacks on Vision Transformers with Token Gradient Regularization

TL;DR

This paper introduces Token Gradient Regularization (TGR), a novel method for transfer-based adversarial attacks on Vision Transformers, which reduces gradient variance in internal blocks to improve attack transferability and effectiveness.

Contribution

The paper proposes TGR, a new gradient regularization technique tailored for ViTs that enhances transfer-based attack success by stabilizing gradients across internal blocks.

Findings

TGR improves attack transferability by 8.8% on average over state-of-the-art methods.

Extensive experiments confirm TGR's effectiveness against ViTs and CNNs.

TGR reduces gradient variance in internal blocks, leading to more effective adversarial samples.

Abstract

Vision transformers (ViTs) have been successfully deployed in a variety of computer vision tasks, but they are still vulnerable to adversarial samples. Transfer-based attacks use a local model to generate adversarial samples and directly transfer them to attack a target black-box model. The high efficiency of transfer-based attacks makes it a severe security threat to ViT-based applications. Therefore, it is vital to design effective transfer-based attacks to identify the deficiencies of ViTs beforehand in security-sensitive scenarios. Existing efforts generally focus on regularizing the input gradients to stabilize the updated direction of adversarial samples. However, the variance of the back-propagated gradients in intermediate blocks of ViTs may still be large, which may make the generated adversarial samples focus on some model-specific features and get stuck in poor local optima. To overcome the shortcomings of existing approaches, we propose the Token Gradient Regularization (TGR) method. According to the structural characteristics of ViTs, TGR reduces the variance of the back-propagated gradient in each internal block of ViTs in a token-wise manner and utilizes the regularized gradient to generate adversarial samples. Extensive experiments on attacking both ViTs and CNNs confirm the superiority of our approach. Notably, compared to the state-of-the-art transfer-based attacks, our TGR offers a performance improvement of 8.8% on average.