Robust Linear Regression: Optimal Rates in Polynomial Time

TL;DR

This paper introduces a computationally efficient robust linear regression estimator that achieves statistically optimal convergence rates under minimal distributional assumptions, resolving a key open problem in the field.

Contribution

It presents the first polynomial-time estimator that attains optimal convergence rates for robust linear regression under hypercontractive distributions with adversarial corruption.

Findings

Achieves optimal convergence rate proportional to ^{2-2/k} under hypercontractive assumptions.

Provides a polynomial-time algorithm exploiting independence via sum-of-squares methods.

Matches the information-theoretic lower bounds for robust linear regression.

Abstract

We obtain robust and computationally efficient estimators for learning several linear models that achieve statistically optimal convergence rate under minimal distributional assumptions. Concretely, we assume our data is drawn from a $k$-hypercontractive distribution and an $\epsilon$-fraction is adversarially corrupted. We then describe an estimator that converges to the optimal least-squares minimizer for the true distribution at a rate proportional to $\epsilon^{2-2/k}$, when the noise is independent of the covariates. We note that no such estimator was known prior to our work, even with access to unbounded computation. The rate we achieve is information-theoretically optimal and thus we resolve the main open question in Klivans, Kothari and Meka [COLT'18]. Our key insight is to identify an analytic condition that serves as a polynomial relaxation of independence of random variables. In particular, we show that when the moments of the noise and covariates are negatively-correlated, we obtain the same rate as independent noise. Further, when the condition is not satisfied, we obtain a rate proportional to $\epsilon^{2-4/k}$, and again match the information-theoretic lower bound. Our central technical contribution is to algorithmically exploit independence of random variables in the "sum-of-squares" framework by formulating it as the aforementioned polynomial inequality.