Lower bounds on the performance of polynomial-time algorithms for sparse linear regression

TL;DR

This paper establishes a fundamental computational barrier in sparse linear regression, showing that polynomial-time algorithms cannot achieve the same prediction accuracy as optimal algorithms under certain conditions, assuming standard complexity conjectures.

Contribution

It proves the first known lower bounds on the prediction performance gap between polynomial-time and optimal algorithms for sparse linear regression, independent of average-case complexity assumptions.

Findings

Polynomial-time algorithms have significantly higher prediction error than optimal algorithms in certain cases.

The gap is especially pronounced when the design matrix is ill-conditioned.

This work provides the first complexity-based lower bounds for sparse linear regression.

Abstract

Under a standard assumption in complexity theory (NP not in P/poly), we demonstrate a gap between the minimax prediction risk for sparse linear regression that can be achieved by polynomial-time algorithms, and that achieved by optimal algorithms. In particular, when the design matrix is ill-conditioned, the minimax prediction loss achievable by polynomial-time algorithms can be substantially greater than that of an optimal algorithm. This result is the first known gap between polynomial and optimal algorithms for sparse linear regression, and does not depend on conjectures in average-case complexity.