On efficient robust regression with subquadratic samples

TL;DR

This paper presents a near-linear-time algorithm for robust linear regression with Gaussian covariates, achieving improved prediction error bounds and analyzing fundamental trade-offs among sample complexity, condition number, and efficiency.

Contribution

It introduces a new efficient algorithm with near-linear runtime that improves prediction error bounds and provides lower bounds illustrating the limits of efficient algorithms.

Findings

Algorithm uses $ ilde{O}(d/ ext{epsilon}^4)$ samples and achieves $O( oot ext{sqrt}( ext{epsilon} ext{kappa}))$ error.

SQ lower bounds show that achieving error below $O( oot ext{sqrt}( ext{epsilon} ext{kappa}))$ requires $ ext{Omega}(d^2)$ samples.

Polynomial lower bounds suggest that without certain assumptions, algorithms need significantly more samples to outperform trivial estimators.

Abstract

We revisit the problem of robust linear regression under Gaussian covariates with an unknown covariance matrix of condition number $\kappa$. For this fundamental problem, significant gaps remain in our understanding of the trade-offs among sample complexity, condition number, runtime, and prediction error for efficient algorithms. Our first result is a near-linear-time algorithm that uses $\widetilde{O}(d/\epsilon^4)$ samples, where $d$ is the dimension and $\epsilon$ is the corruption rate, and achieves prediction error $O(\sqrt{\epsilon\kappa})$ under the condition $\epsilon\kappa \lesssim 1$, improving over all prior works. We complement this result with a Statistical Query (SQ) lower bound showing that efficient SQ algorithms achieving error $o(\sqrt{\epsilon\kappa})$ when $\epsilon \kappa \lesssim 1$ require queries that take $\Omega(d^2)$ samples to simulate. Finally, we prove a low-degree polynomial lower bound that gives fine-grained evidence that, without assumptions such as $\epsilon \kappa \lesssim 1$, efficient algorithms may require $\tilde{\Omega}\left(\min\{d\epsilon^{2}\kappa^{2},\ \epsilon^{2}d^{2}\}\right)$ samples to significantly outperform the trivial estimator that always guesses $0$.