On the Geometry of Regularization in Adversarial Training: High-Dimensional Asymptotics and Generalization Bounds

TL;DR

This paper provides a detailed high-dimensional asymptotic analysis of regularization in adversarial training for binary classification, offering insights into optimal regularization choices and generalization bounds under various attack scenarios.

Contribution

It derives exact asymptotic descriptions of robust regularized empirical risk minimizers and bounds on Rademacher complexity for high-dimensional adversarial training.

Findings

Optimal regularization norms depend on perturbation size.

Regularization becomes more critical as perturbations increase.

Theoretical bounds inform regularization choices in scarce data regimes.

Abstract

Regularization, whether explicit in terms of a penalty in the loss or implicit in the choice of algorithm, is a cornerstone of modern machine learning. Indeed, controlling the complexity of the model class is particularly important when data is scarce, noisy or contaminated, as it translates a statistical belief on the underlying structure of the data. This work investigates the question of how to choose the regularization norm $\lVert \cdot \rVert$ in the context of high-dimensional adversarial training for binary classification. To this end, we first derive an exact asymptotic description of the robust, regularized empirical risk minimizer for various types of adversarial attacks and regularization norms (including non-$\ell_p$ norms). We complement this analysis with a uniform convergence analysis, deriving bounds on the Rademacher Complexity for this class of problems. Leveraging our theoretical results, we quantitatively characterize the relationship between perturbation size and the optimal choice of $\lVert \cdot \rVert$, confirming the intuition that, in the data scarce regime, the type of regularization becomes increasingly important for adversarial training as perturbations grow in size.