Towards Robust Vision Transformer via Masked Adaptive Ensemble

TL;DR

This paper introduces a novel Vision Transformer architecture with an adaptive ensemble and detection mechanism to improve robustness against adversarial attacks while maintaining high standard accuracy.

Contribution

It proposes a new ViT design with a detector and adaptive ensemble, enhancing robustness and accuracy trade-offs, and introduces a patch masking technique for better defense against adaptive attacks.

Findings

Achieves 90.3% standard accuracy on CIFAR-10.

Attains 49.8% adversarial robustness against attacks.

Outperforms existing methods in robustness and accuracy trade-offs.

Abstract

Adversarial training (AT) can help improve the robustness of Vision Transformers (ViT) against adversarial attacks by intentionally injecting adversarial examples into the training data. However, this way of adversarial injection inevitably incurs standard accuracy degradation to some extent, thereby calling for a trade-off between standard accuracy and robustness. Besides, the prominent AT solutions are still vulnerable to adaptive attacks. To tackle such shortcomings, this paper proposes a novel ViT architecture, including a detector and a classifier bridged by our newly developed adaptive ensemble. Specifically, we empirically discover that detecting adversarial examples can benefit from the Guided Backpropagation technique. Driven by this discovery, a novel Multi-head Self-Attention (MSA) mechanism is introduced to enhance our detector to sniff adversarial examples. Then, a classifier with two encoders is employed for extracting visual representations respectively from clean images and adversarial examples, with our adaptive ensemble to adaptively adjust the proportion of visual representations from the two encoders for accurate classification. This design enables our ViT architecture to achieve a better trade-off between standard accuracy and robustness. Besides, our adaptive ensemble technique allows us to mask off a random subset of image patches within input data, boosting our ViT's robustness against adaptive attacks, while maintaining high standard accuracy. Experimental results exhibit that our ViT architecture, on CIFAR-10, achieves the best standard accuracy and adversarial robustness of 90.3% and 49.8%, respectively.