Unlabeled Data Improves Adversarial Robustness

TL;DR

This paper shows that unlabeled data can significantly improve adversarial robustness through semisupervised learning, both theoretically by bridging sample complexity gaps and empirically by enhancing robustness on CIFAR-10 and SVHN datasets.

Contribution

It provides a theoretical proof that unlabeled data bridges the robustness gap and demonstrates empirically that semisupervised learning improves adversarial robustness on multiple datasets.

Findings

Unlabeled data bridges the sample complexity gap for robust classification.

Self-training with unlabeled data achieves high robust accuracy with fewer labels.

Augmenting datasets with unlabeled data improves robustness against strong attacks.

Abstract

We demonstrate, theoretically and empirically, that adversarial robustness can significantly benefit from semisupervised learning. Theoretically, we revisit the simple Gaussian model of Schmidt et al. that shows a sample complexity gap between standard and robust classification. We prove that unlabeled data bridges this gap: a simple semisupervised learning procedure (self-training) achieves high robust accuracy using the same number of labels required for achieving high standard accuracy. Empirically, we augment CIFAR-10 with 500K unlabeled images sourced from 80 Million Tiny Images and use robust self-training to outperform state-of-the-art robust accuracies by over 5 points in (i) $\ell_\infty$ robustness against several strong attacks via adversarial training and (ii) certified $\ell_2$ and $\ell_\infty$ robustness via randomized smoothing. On SVHN, adding the dataset's own extra training set with the labels removed provides gains of 4 to 10 points, within 1 point of the gain from using the extra labels.