On cross-validated Lasso in high dimensions

TL;DR

This paper provides theoretical guarantees for the cross-validated Lasso estimator in high-dimensional settings, showing it achieves near-optimal convergence rates even with non-Gaussian noise and when the number of covariates exceeds the sample size.

Contribution

It offers the first non-asymptotic error bounds for the cross-validated Lasso, demonstrating its near-optimal convergence rates in high-dimensional, non-Gaussian noise scenarios.

Findings

Cross-validated Lasso converges at near-optimal rates in prediction, $L^2$, and $L^1$ norms.

Results hold even when the number of covariates exceeds the sample size.

Theoretical justification for using cross-validation to select the Lasso penalty parameter.

Abstract

In this paper, we derive non-asymptotic error bounds for the Lasso estimator when the penalty parameter for the estimator is chosen using $K$-fold cross-validation. Our bounds imply that the cross-validated Lasso estimator has nearly optimal rates of convergence in the prediction, $L^2$, and $L^1$ norms. For example, we show that in the model with the Gaussian noise and under fairly general assumptions on the candidate set of values of the penalty parameter, the estimation error of the cross-validated Lasso estimator converges to zero in the prediction norm with the $\sqrt{s\log p / n}\times \sqrt{\log(p n)}$ rate, where $n$ is the sample size of available data, $p$ is the number of covariates, and $s$ is the number of non-zero coefficients in the model. Thus, the cross-validated Lasso estimator achieves the fastest possible rate of convergence in the prediction norm up to a small logarithmic factor $\sqrt{\log(p n)}$, and similar conclusions apply for the convergence rate both in $L^2$ and in $L^1$ norms. Importantly, our results cover the case when $p$ is (potentially much) larger than $n$ and also allow for the case of non-Gaussian noise. Our paper therefore serves as a justification for the widely spread practice of using cross-validation as a method to choose the penalty parameter for the Lasso estimator.