Why adversarial training can hurt robust accuracy

TL;DR

This paper reveals that adversarial training, commonly thought to improve robustness, can actually harm robust accuracy in small data regimes, supported by theoretical proofs and experiments on image datasets.

Contribution

It demonstrates that adversarial training may decrease robust generalization in low-data settings, providing theoretical insights and empirical evidence.

Findings

Adversarial training can hurt robustness with limited data

Theoretical proof in high-dimensional linear models

Experimental validation on image datasets with perceptible attacks

Abstract

Machine learning classifiers with high test accuracy often perform poorly under adversarial attacks. It is commonly believed that adversarial training alleviates this issue. In this paper, we demonstrate that, surprisingly, the opposite may be true -- Even though adversarial training helps when enough data is available, it may hurt robust generalization in the small sample size regime. We first prove this phenomenon for a high-dimensional linear classification setting with noiseless observations. Our proof provides explanatory insights that may also transfer to feature learning models. Further, we observe in experiments on standard image datasets that the same behavior occurs for perceptible attacks that effectively reduce class information such as mask attacks and object corruptions.