A Fundamental Accuracy--Robustness Trade-off in Regression and Classification

TL;DR

This paper establishes a fundamental trade-off between accuracy and adversarial robustness in machine learning models, showing that achieving robustness often reduces accuracy unless certain regularity conditions are met.

Contribution

The paper formalizes a general trade-off between standard and adversarial risk and provides a necessary condition for robustness without accuracy loss, based on data distribution properties.

Findings

Derived a universal trade-off between accuracy and robustness.

Evaluated the trade-off in polynomial ridge regression.

Identified a data distribution condition for robustness without accuracy loss.

Abstract

We derive a fundamental trade-off between standard and adversarial risk in a rather general situation that formalizes the following simple intuition: "If no (nearly) optimal predictor is smooth, adversarial robustness comes at the cost of accuracy." As a concrete example, we evaluate the derived trade-off in regression with polynomial ridge functions under mild regularity conditions. Generalizing our analysis of this example, we formulate a necessary condition under which adversarial robustness can be achieved without significant degradation of the accuracy. This necessary condition is expressed in terms of a quantity that resembles the Poincar\'{e} constant of the data distribution.