Robust Linear Regression: Phase-Transitions and Precise Tradeoffs for General Norms

TL;DR

This paper explores the fundamental tradeoffs between adversarial robustness and predictive accuracy in linear regression models, providing precise characterizations across various regimes and attack norms.

Contribution

It offers a comprehensive theoretical analysis of robustness-accuracy tradeoffs in linear regression under adversarial attacks, extending previous work to general norms and settings.

Findings

Identifies regimes where robustness does not harm accuracy

Characterizes unavoidable tradeoffs in certain regimes

Empirically validates theoretical predictions

Abstract

In this paper, we investigate the impact of test-time adversarial attacks on linear regression models and determine the optimal level of robustness that any model can reach while maintaining a given level of standard predictive performance (accuracy). Through quantitative estimates, we uncover fundamental tradeoffs between adversarial robustness and accuracy in different regimes. We obtain a precise characterization which distinguishes between regimes where robustness is achievable without hurting standard accuracy and regimes where a tradeoff might be unavoidable. Our findings are empirically confirmed with simple experiments that represent a variety of settings. This work applies to feature covariance matrices and attack norms of any nature, and extends beyond previous works in this area.