Computational Complexity versus Statistical Performance on Sparse Recovery Problems

TL;DR

This paper reveals that key parameters governing compressed sensing performance also determine the computational complexity of algorithms, linking statistical recovery guarantees with algorithmic efficiency through classical condition numbers.

Contribution

It establishes a direct connection between classical parameters controlling recovery performance and those controlling algorithmic complexity in compressed sensing.

Findings

Sharpness at the optimum controls convergence speed.

The ratio of true sparsity to recoverable signal size defines a condition number.

Worst-case complexity measure matches the restricted singular value.

Abstract

We show that several classical quantities controlling compressed sensing performance directly match classical parameters controlling algorithmic complexity. We first describe linearly convergent restart schemes on first-order methods solving a broad range of compressed sensing problems, where sharpness at the optimum controls convergence speed. We show that for sparse recovery problems, this sharpness can be written as a condition number, given by the ratio between true signal sparsity and the largest signal size that can be recovered by the observation matrix. In a similar vein, Renegar's condition number is a data-driven complexity measure for convex programs, generalizing classical condition numbers for linear systems. We show that for a broad class of compressed sensing problems, the worst case value of this algorithmic complexity measure taken over all signals matches the restricted singular value of the observation matrix which controls robust recovery performance. Overall, this means in both cases that, in compressed sensing problems, a single parameter directly controls both computational complexity and recovery performance. Numerical experiments illustrate these points using several classical algorithms.