Robustness and Accuracy Could Be Reconcilable by (Proper) Definition

TL;DR

This paper introduces SCORE, a new robust error measure that reconciles robustness and accuracy in adversarial training by redefining the concept of model invariance, leading to improved performance and insights.

Contribution

The paper proposes SCORE, a self-consistent robust error measure based on local equivariance, enabling better robustness-accuracy trade-offs and efficient optimization.

Findings

Models using SCORE achieve top performance on RobustBench.

SCORE provides insights into overfitting and semantic gradients in robust models.

Efficient minimization of SCORE is possible with simple metric substitutions.

Abstract

The trade-off between robustness and accuracy has been widely studied in the adversarial literature. Although still controversial, the prevailing view is that this trade-off is inherent, either empirically or theoretically. Thus, we dig for the origin of this trade-off in adversarial training and find that it may stem from the improperly defined robust error, which imposes an inductive bias of local invariance -- an overcorrection towards smoothness. Given this, we advocate employing local equivariance to describe the ideal behavior of a robust model, leading to a self-consistent robust error named SCORE. By definition, SCORE facilitates the reconciliation between robustness and accuracy, while still handling the worst-case uncertainty via robust optimization. By simply substituting KL divergence with variants of distance metrics, SCORE can be efficiently minimized. Empirically, our models achieve top-rank performance on RobustBench under AutoAttack. Besides, SCORE provides instructive insights for explaining the overfitting phenomenon and semantic input gradients observed on robust models. Code is available at https://github.com/P2333/SCORE.