Adversarial Training with Generated Data in High-Dimensional Regression: An Asymptotic Study

TL;DR

This paper provides a theoretical analysis of adversarial training with generated data in high-dimensional linear regression, revealing improved performance with regularization and offering a practical cross-validation shortcut.

Contribution

It offers the first asymptotic analysis of two-stage adversarial training with generated data in high-dimensional settings, highlighting the benefits of regularization.

Findings

Double-descent phenomenon observed in ridgeless training.

Regularization improves adversarial training performance.

Derived a shortcut cross-validation formula for the method.

Abstract

In recent years, studies such as \cite{carmon2019unlabeled,gowal2021improving,xing2022artificial} have demonstrated that incorporating additional real or generated data with pseudo-labels can enhance adversarial training through a two-stage training approach. In this paper, we perform a theoretical analysis of the asymptotic behavior of this method in high-dimensional linear regression. While a double-descent phenomenon can be observed in ridgeless training, with an appropriate $\mathcal{L}_2$ regularization, the two-stage adversarial training achieves a better performance. Finally, we derive a shortcut cross-validation formula specifically tailored for the two-stage training method.