On the best choice of Lasso program given data parameters

TL;DR

This paper analyzes the parameter sensitivity of generalized Lasso programs in generalized compressed sensing, revealing how different constraints affect risk behavior and guiding practitioners in choosing the appropriate program.

Contribution

It provides theoretical insights into the asymptotic risk behavior and parameter sensitivity of three GL programs, highlighting differences and practical implications.

Findings

Gauge-constrained GL exhibits cusp-like risk behavior in low-noise limit.

Residual-constrained Lasso has suboptimal risk for very sparse signals.

Unconstrained Lasso is less sensitive to parameter variations.

Abstract

Generalized compressed sensing (GCS) is a paradigm in which a structured high-dimensional signal may be recovered from random, under-determined, and corrupted linear measurements. Generalized Lasso (GL) programs are effective for solving GCS problems due to their proven ability to leverage underlying signal structure. Three popular GL programs are equivalent in a sense and sometimes used interchangeably. Tuned by a governing parameter, each admit an optimal parameter choice. For sparse or low-rank signal structures, this choice yields minimax order-optimal error. While GCS is well-studied, existing theory for GL programs typically concerns this optimally tuned setting. However, the optimal parameter value for a GL program depends on properties of the data, and is typically unknown in practical settings. Performance in empirical problems thus hinges on a program's parameter sensitivity: it is desirable that small variation about the optimal parameter choice begets small variation about the optimal risk. We examine the risk for these three programs and demonstrate that their parameter sensitivity can differ for the same data. We prove a gauge-constrained GL program admits asymptotic cusp-like behaviour of its risk in the limiting low-noise regime. We prove that a residual-constrained Lasso program has asymptotically suboptimal risk for very sparse vectors. These results contrast observations about an unconstrained Lasso program, which is relatively less sensitive to its parameter choice. We support the asymptotic theory with numerical simulations, demonstrating that parameter sensitivity of GL programs is readily observed for even modest dimensional parameters. Importantly, these simulations demonstrate regimes in which a GL program exhibits sensitivity to its parameter choice, though the other two do not. We hope this work aids practitioners in selecting a GL program for their problem.