Robust Learning Meets Generative Models: Can Proxy Distributions Improve Adversarial Robustness?

TL;DR

This paper explores how using proxy distributions generated by advanced models can enhance adversarial robustness in neural networks, providing theoretical bounds and empirical improvements across multiple datasets.

Contribution

It introduces a formal analysis linking robustness transfer to Wasserstein distance and demonstrates significant empirical gains using generative model-based proxy distributions.

Findings

Robust accuracy improved by up to 7.5% on CIFAR-10.

Theoretical bound relates robustness difference to Wasserstein distance.

Diffusion models outperform GANs as proxy distributions.

Abstract

While additional training data improves the robustness of deep neural networks against adversarial examples, it presents the challenge of curating a large number of specific real-world samples. We circumvent this challenge by using additional data from proxy distributions learned by advanced generative models. We first seek to formally understand the transfer of robustness from classifiers trained on proxy distributions to the real data distribution. We prove that the difference between the robustness of a classifier on the two distributions is upper bounded by the conditional Wasserstein distance between them. Next we use proxy distributions to significantly improve the performance of adversarial training on five different datasets. For example, we improve robust accuracy by up to 7.5% and 6.7% in $\ell_{\infty}$ and $\ell_2$ threat model over baselines that are not using proxy distributions on the CIFAR-10 dataset. We also improve certified robust accuracy by 7.6% on the CIFAR-10 dataset. We further demonstrate that different generative models bring a disparate improvement in the performance in robust training. We propose a robust discrimination approach to characterize the impact of individual generative models and further provide a deeper understanding of why current state-of-the-art in diffusion-based generative models are a better choice for proxy distribution than generative adversarial networks.