Robust Linear Regression: Gradient-descent, Early-stopping, and Beyond

TL;DR

This paper investigates the robustness of gradient descent with early stopping in linear regression against various adversarial attacks, proposing data transformations and a new estimator to enhance robustness across attack types.

Contribution

It introduces a data transformation approach for gradient descent to improve robustness against Mahalanobis attacks and proposes a simple estimator with near-optimal adversarial risk.

Findings

Early-stopped GD is optimally robust against Euclidean attacks.

Data transformations enable GD to handle general Mahalanobis attacks.

A simple estimator achieves near-optimal adversarial risk across norms.

Abstract

In this work we study the robustness to adversarial attacks, of early-stopping strategies on gradient-descent (GD) methods for linear regression. More precisely, we show that early-stopped GD is optimally robust (up to an absolute constant) against Euclidean-norm adversarial attacks. However, we show that this strategy can be arbitrarily sub-optimal in the case of general Mahalanobis attacks. This observation is compatible with recent findings in the case of classification~\cite{Vardi2022GradientMP} that show that GD provably converges to non-robust models. To alleviate this issue, we propose to apply instead a GD scheme on a transformation of the data adapted to the attack. This data transformation amounts to apply feature-depending learning rates and we show that this modified GD is able to handle any Mahalanobis attack, as well as more general attacks under some conditions. Unfortunately, choosing such adapted transformations can be hard for general attacks. To the rescue, we design a simple and tractable estimator whose adversarial risk is optimal up to within a multiplicative constant of 1.1124 in the population regime, and works for any norm.