Robust Regression and Lasso

TL;DR

This paper reveals that Lasso's sparsity and robustness properties are interconnected, providing new insights into its behavior, stability, and generalizations through a robust optimization perspective.

Contribution

It demonstrates that Lasso's solution can be viewed as a robust optimization problem, linking sparsity, noise protection, and stability, and introduces new geometric and theoretical insights.

Findings

Lasso solutions are robust to noise, explaining their sparsity.

Robust optimization provides a new perspective on Lasso's properties.

Lasso is shown to be inconsistent and not stable due to sparsity-stability trade-off.

Abstract

Lasso, or $\ell^1$ regularized least squares, has been explored extensively for its remarkable sparsity properties. It is shown in this paper that the solution to Lasso, in addition to its sparsity, has robustness properties: it is the solution to a robust optimization problem. This has two important consequences. First, robustness provides a connection of the regularizer to a physical property, namely, protection from noise. This allows a principled selection of the regularizer, and in particular, generalizations of Lasso that also yield convex optimization problems are obtained by considering different uncertainty sets. Secondly, robustness can itself be used as an avenue to exploring different properties of the solution. In particular, it is shown that robustness of the solution explains why the solution is sparse. The analysis as well as the specific results obtained differ from standard sparsity results, providing different geometric intuition. Furthermore, it is shown that the robust optimization formulation is related to kernel density estimation, and based on this approach, a proof that Lasso is consistent is given using robustness directly. Finally, a theorem saying that sparsity and algorithmic stability contradict each other, and hence Lasso is not stable, is presented.