Distributional Hardness Against Preconditioned Lasso via Erasure-Robust Designs

TL;DR

This paper establishes a fundamental hardness result for preconditioned Lasso in sparse linear regression, demonstrating that no single preconditioning strategy can reliably recover certain signals without a linear number of samples, supported by new compressed sensing insights.

Contribution

It proves a strong lower bound showing that all invertibly-preconditioned Lasso programs fail on a specific signal distribution unless given many samples, and introduces a novel robustness result in compressed sensing under erasures.

Findings

Any invertibly-preconditioned Lasso fails on a specific signal with high probability unless given linear samples.

Standard sparse random designs are robust to measurement erasures, with most signal coordinates still identifiable.

First study of partial recoverability of sparse signals under erasures in compressed sensing.

Abstract

Sparse linear regression with ill-conditioned Gaussian random designs is widely believed to exhibit a statistical/computational gap, but there is surprisingly little formal evidence for this belief, even in the form of examples that are hard for restricted classes of algorithms. Recent work has shown that, for certain covariance matrices, the broad class of Preconditioned Lasso programs provably cannot succeed on polylogarithmically sparse signals with a sublinear number of samples. However, this lower bound only shows that for every preconditioner, there exists at least one signal that it fails to recover successfully. This leaves open the possibility that, for example, trying multiple different preconditioners solves every sparse linear regression problem. In this work, we prove a stronger lower bound that overcomes this issue. For an appropriate covariance matrix, we construct a single signal distribution on which any invertibly-preconditioned Lasso program fails with high probability, unless it receives a linear number of samples. Surprisingly, at the heart of our lower bound is a new positive result in compressed sensing. We show that standard sparse random designs are with high probability robust to adversarial measurement erasures, in the sense that if $b$ measurements are erased, then all but $O(b)$ of the coordinates of the signal are still information-theoretically identifiable. To our knowledge, this is the first time that partial recoverability of arbitrary sparse signals under erasures has been studied in compressed sensing.