Towards Practical Certifiable Patch Defense with Vision Transformer

TL;DR

This paper introduces a Vision Transformer-based framework with a novel training task and attention structure modifications to achieve high certified robustness against patch attacks while maintaining near-normal accuracy.

Contribution

It proposes a new ViT-based certifiable patch defense with a progressive training task and isolated band self-attention, significantly improving certified and clean accuracy.

Findings

41.70% certified accuracy on ImageNet under 2% patch attack

78.58% clean accuracy close to ResNet-101

State-of-the-art performance on CIFAR-10 and ImageNet

Abstract

Patch attacks, one of the most threatening forms of physical attack in adversarial examples, can lead networks to induce misclassification by modifying pixels arbitrarily in a continuous region. Certifiable patch defense can guarantee robustness that the classifier is not affected by patch attacks. Existing certifiable patch defenses sacrifice the clean accuracy of classifiers and only obtain a low certified accuracy on toy datasets. Furthermore, the clean and certified accuracy of these methods is still significantly lower than the accuracy of normal classification networks, which limits their application in practice. To move towards a practical certifiable patch defense, we introduce Vision Transformer (ViT) into the framework of Derandomized Smoothing (DS). Specifically, we propose a progressive smoothed image modeling task to train Vision Transformer, which can capture the more discriminable local context of an image while preserving the global semantic information. For efficient inference and deployment in the real world, we innovatively reconstruct the global self-attention structure of the original ViT into isolated band unit self-attention. On ImageNet, under 2% area patch attacks our method achieves 41.70% certified accuracy, a nearly 1-fold increase over the previous best method (26.00%). Simultaneously, our method achieves 78.58% clean accuracy, which is quite close to the normal ResNet-101 accuracy. Extensive experiments show that our method obtains state-of-the-art clean and certified accuracy with inferring efficiently on CIFAR-10 and ImageNet.