A Practical Scheme and Fast Algorithm to Tune the Lasso With Optimality Guarantees

TL;DR

This paper presents a new, fast scheme for selecting the regularization parameter in high-dimensional Lasso regression, providing optimal guarantees and outperforming standard methods like cross-validation in speed and accuracy.

Contribution

A novel, Lepski-inspired scheme for tuning Lasso with finite-sample optimality guarantees and a simple, fast implementation along a single Lasso path.

Findings

Scheme matches oracle performance up to a small constant

Method is faster and more accurate than cross-validation

Applicable to simulated and real data

Abstract

We introduce a novel scheme for choosing the regularization parameter in high-dimensional linear regression with Lasso. This scheme, inspired by Lepski's method for bandwidth selection in non-parametric regression, is equipped with both optimal finite-sample guarantees and a fast algorithm. In particular, for any design matrix such that the Lasso has low sup-norm error under an "oracle choice" of the regularization parameter, we show that our method matches the oracle performance up to a small constant factor, and show that it can be implemented by performing simple tests along a single Lasso path. By applying the Lasso to simulated and real data, we find that our novel scheme can be faster and more accurate than standard schemes such as Cross-Validation.