The Limitations of Adversarial Training and the Blind-Spot Attack

TL;DR

This paper reveals that adversarial training's robustness is limited by the data manifold and introduces the 'blind-spot attack' targeting low-density regions, exposing vulnerabilities in both empirical and provable defenses, especially in high-dimensional datasets.

Contribution

The paper identifies the existence of blind-spots in adversarial training and provable defenses, demonstrating their impact on robustness in high-dimensional data.

Findings

Blind-spots can be easily found in MNIST by simple transformations.

Blind-spots pose significant challenges for defending high-dimensional datasets like CIFAR and ImageNet.

Provable defenses are also susceptible to blind-spot attacks due to limited robustness certificates.

Abstract

The adversarial training procedure proposed by Madry et al. (2018) is one of the most effective methods to defend against adversarial examples in deep neural networks (DNNs). In our paper, we shed some lights on the practicality and the hardness of adversarial training by showing that the effectiveness (robustness on test set) of adversarial training has a strong correlation with the distance between a test point and the manifold of training data embedded by the network. Test examples that are relatively far away from this manifold are more likely to be vulnerable to adversarial attacks. Consequentially, an adversarial training based defense is susceptible to a new class of attacks, the "blind-spot attack", where the input images reside in "blind-spots" (low density regions) of the empirical distribution of training data but is still on the ground-truth data manifold. For MNIST, we found that these blind-spots can be easily found by simply scaling and shifting image pixel values. Most importantly, for large datasets with high dimensional and complex data manifold (CIFAR, ImageNet, etc), the existence of blind-spots in adversarial training makes defending on any valid test examples difficult due to the curse of dimensionality and the scarcity of training data. Additionally, we find that blind-spots also exist on provable defenses including (Wong & Kolter, 2018) and (Sinha et al., 2018) because these trainable robustness certificates can only be practically optimized on a limited set of training data.