SlowFormer: Universal Adversarial Patch for Attack on Compute and Energy Efficiency of Inference Efficient Vision Transformers

TL;DR

This paper introduces a universal adversarial patch attack targeting efficient vision transformers, significantly increasing their compute and energy consumption, and discusses potential defenses against such attacks.

Contribution

It reveals vulnerability of adaptive efficient vision transformers to a universal patch attack that can maximize computation with minimal patch size.

Findings

Attack can increase computation to maximum with only 8% patch.

Standard adversarial training reduces attack success.

Efficient models are vulnerable to universal adversarial patches.

Abstract

Recently, there has been a lot of progress in reducing the computation of deep models at inference time. These methods can reduce both the computational needs and power usage of deep models. Some of these approaches adaptively scale the compute based on the input instance. We show that such models can be vulnerable to a universal adversarial patch attack, where the attacker optimizes for a patch that when pasted on any image, can increase the compute and power consumption of the model. We run experiments with three different efficient vision transformer methods showing that in some cases, the attacker can increase the computation to the maximum possible level by simply pasting a patch that occupies only 8\% of the image area. We also show that a standard adversarial training defense method can reduce some of the attack's success. We believe adaptive efficient methods will be necessary for the future to lower the power usage of deep models, so we hope our paper encourages the community to study the robustness of these methods and develop better defense methods for the proposed attack.