Hedging parameter selection for basis pursuit

TL;DR

This paper introduces a new hyper-parameter selection method for LASSO in compressed sensing, combining Hedge online learning with stochastic Frank-Wolfe, achieving comparable prediction accuracy to cross-validation with less computation.

Contribution

It proposes a novel hyper-parameter selection approach for LASSO using Hedge and Frank-Wolfe methods, reducing computational cost while maintaining accuracy.

Findings

Achieves prediction performance comparable to cross-validation.

Reduces computational cost of hyper-parameter tuning.

Demonstrates effectiveness in high-dimensional estimation.

Abstract

In Compressed Sensing and high dimensional estimation, signal recovery often relies on sparsity assumptions and estimation is performed via $\ell_1$-penalized least-squares optimization, a.k.a. LASSO. The $\ell_1$ penalisation is usually controlled by a weight, also called "relaxation parameter", denoted by $\lambda$. It is commonly thought that the practical efficiency of the LASSO for prediction crucially relies on accurate selection of $\lambda$. In this short note, we propose to consider the hyper-parameter selection problem from a new perspective which combines the Hedge online learning method by Freund and Shapire, with the stochastic Frank-Wolfe method for the LASSO. Using the Hedge algorithm, we show that a our simple selection rule can achieve prediction results comparable to Cross Validation at a potentially much lower computational cost.