Asymptotic Theory of $K$-fold Cross-validation in Lasso and the validity of Bootstrap

TL;DR

This paper provides a theoretical foundation for the use of K-fold cross-validation in Lasso regression, demonstrating its consistency and validating bootstrap methods for inference.

Contribution

It establishes the asymptotic properties of K-fold CV Lasso and confirms bootstrap validity for statistical inference in heteroscedastic linear models.

Findings

K-fold CV Lasso is n^{1/2}-consistent under certain conditions.

Bootstrap approximates the distribution of K-fold CV Lasso estimator accurately.

Bootstrap-based inference is validated through simulations and real data.

Abstract

Least absolute shrinkage and selection operator or Lasso is one of the widely used regularization methods in regression. Statisticians usually implement Lasso in practice by choosing the penalty parameter in a data-dependent way, the most popular being the $K-$fold cross-validation (or $K-$fold CV). However, inferential properties, such as the variable selection consistency and $n^{1/2}-$consistency, of the $K-$fold CV based Lasso estimator and validity of the Bootstrap approximation are still unknown. In this paper, we consider the heteroscedastic linear regression model and show only under some moment type conditions that the Lasso estimator with $K$-fold CV based penalty is $n^{1/2}-$consistent, but not variable selection consistent. Additionally, we establish the validity of Bootstrap in approximating the distribution of the $K-$fold CV based Lasso estimator. Therefore, our results theoretically justify the use of $K-$fold CV based Lasso estimator to perform statistical inference in linear regression. We validate our Bootstrap method for the $K-$fold CV based Lasso estimator in finite samples based on simulations. We also implement our Bootstrap based inference on a real data set.