Backdoor Attack Against Vision Transformers via Attention Gradient-Based Image Erosion

TL;DR

This paper introduces AGEB, a novel backdoor attack on Vision Transformers that balances stealthiness and effectiveness by eroding pixels in high-attention areas, maintaining model accuracy and invisibility of triggers.

Contribution

The paper proposes AGEB, a new backdoor attack method for ViTs that leverages attention gradients to embed covert triggers without compromising clean accuracy.

Findings

Achieves high attack success rate across ViT architectures

Maintains model accuracy on clean samples

Ensures minimal visual differences between clean and triggered images

Abstract

Vision Transformers (ViTs) have outperformed traditional Convolutional Neural Networks (CNN) across various computer vision tasks. However, akin to CNN, ViTs are vulnerable to backdoor attacks, where the adversary embeds the backdoor into the victim model, causing it to make wrong predictions about testing samples containing a specific trigger. Existing backdoor attacks against ViTs have the limitation of failing to strike an optimal balance between attack stealthiness and attack effectiveness. In this work, we propose an Attention Gradient-based Erosion Backdoor (AGEB) targeted at ViTs. Considering the attention mechanism of ViTs, AGEB selectively erodes pixels in areas of maximal attention gradient, embedding a covert backdoor trigger. Unlike previous backdoor attacks against ViTs, AGEB achieves an optimal balance between attack stealthiness and attack effectiveness, ensuring the trigger remains invisible to human detection while preserving the model's accuracy on clean samples. Extensive experimental evaluations across various ViT architectures and datasets confirm the effectiveness of AGEB, achieving a remarkable Attack Success Rate (ASR) without diminishing Clean Data Accuracy (CDA). Furthermore, the stealthiness of AGEB is rigorously validated, demonstrating minimal visual discrepancies between the clean and the triggered images.