Robust Sparse Regression with Non-Isotropic Designs

TL;DR

This paper introduces robust polynomial-time algorithms for sparse linear regression under non-isotropic designs and adversarial noise, achieving near-optimal error bounds with broad distributional assumptions.

Contribution

It develops new algorithms using sum-of-squares relaxations and Huber loss minimization, handling heavy-tailed, non-isotropic designs with adversarial contamination, and provides statistical query lower bounds.

Findings

Achieves error $O( oot{rac{ ext{error}}{ ext{sample size}}})$ under certain moment conditions.

First polynomial-time algorithms with error $o( oot{ ext{error}})$ in sparse regression with non-isotropic designs.

Provides nearly matching statistical query lower bounds for the problem.

Abstract

We develop a technique to design efficiently computable estimators for sparse linear regression in the simultaneous presence of two adversaries: oblivious and adaptive. We design several robust algorithms that outperform the state of the art even in the special case when oblivious adversary simply adds Gaussian noise. In particular, we provide a polynomial-time algorithm that with high probability recovers the signal up to error $O(\sqrt{\varepsilon})$ as long as the number of samples $n \ge \tilde{O}(k^2/\varepsilon)$, only assuming some bounds on the third and the fourth moments of the distribution ${D}$ of the design. In addition, prior to this work, even in the special case of Gaussian design and noise, no polynomial time algorithm was known to achieve error $o(\sqrt{\varepsilon})$ in the sparse setting $n < d^2$. We show that under some assumptions on the fourth and the eighth moments of ${D}$, there is a polynomial-time algorithm that achieves error $o(\sqrt{\varepsilon})$ as long as $n \ge \tilde{O}(k^4 / \varepsilon^3)$. For Gaussian distribution, this algorithm achieves error $O(\varepsilon^{3/4})$. Moreover, our algorithm achieves error $o(\sqrt{\varepsilon})$ for all log-concave distributions if $\varepsilon \le 1/\text{polylog(d)}$. Our algorithms are based on the filtering of the covariates that uses sum-of-squares relaxations, and weighted Huber loss minimization with $\ell_1$ regularizer. We provide a novel analysis of weighted penalized Huber loss that is suitable for heavy-tailed designs in the presence of two adversaries. Furthermore, we complement our algorithmic results with Statistical Query lower bounds, providing evidence that our estimators are likely to have nearly optimal sample complexity.