The Surprising Harmfulness of Benign Overfitting for Adversarial Robustness

TL;DR

This paper reveals that benign overfitting in large models can be harmful for adversarial robustness, showing a trade-off between standard and adversarial risks in overparameterized models and neural tangent kernel regimes.

Contribution

It provides the first theoretical proof that benign overfitting can increase adversarial vulnerability, highlighting a fundamental trade-off in model robustness.

Findings

Min-norm estimator in overparameterized linear models is adversarially vulnerable.

Trade-off between standard risk and adversarial risk in ridge regression.

Parallel results in neural tangent kernel models align with deep neural network observations.

Abstract

Recent empirical and theoretical studies have established the generalization capabilities of large machine learning models that are trained to (approximately or exactly) fit noisy data. In this work, we prove a surprising result that even if the ground truth itself is robust to adversarial examples, and the benignly overfitted model is benign in terms of the ``standard'' out-of-sample risk objective, this benign overfitting process can be harmful when out-of-sample data are subject to adversarial manipulation. More specifically, our main results contain two parts: (i) the min-norm estimator in overparameterized linear model always leads to adversarial vulnerability in the ``benign overfitting'' setting; (ii) we verify an asymptotic trade-off result between the standard risk and the ``adversarial'' risk of every ridge regression estimator, implying that under suitable conditions these two items cannot both be small at the same time by any single choice of the ridge regularization parameter. Furthermore, under the lazy training regime, we demonstrate parallel results on two-layer neural tangent kernel (NTK) model, which align with empirical observations in deep neural networks. Our finding provides theoretical insights into the puzzling phenomenon observed in practice, where the true target function (e.g., human) is robust against adverasrial attack, while beginly overfitted neural networks lead to models that are not robust.