On the use of cross-validation for the calibration of the adaptive lasso

TL;DR

This paper examines the effectiveness of cross-validation schemes in calibrating the adaptive lasso, demonstrating that simple schemes are suboptimal compared to more elaborate methods, impacting support recovery and prediction accuracy.

Contribution

It clarifies the appropriate cross-validation schemes for adaptive lasso calibration and highlights the shortcomings of simple schemes through empirical examples.

Findings

Simple cross-validation schemes are suboptimal for adaptive lasso calibration.

More elaborate schemes improve support recovery and prediction error.

The study uses synthetic and real-world data to illustrate these points.

Abstract

The adaptive lasso refers to a class of methods that use weighted versions of the $L_1$-norm penalty, with weights derived from an initial estimate of the parameter vector to be estimated. Irrespective of the method chosen to compute this initial estimate, the performance of the adaptive lasso critically depends on the value of a hyperparameter, which controls the magnitude of the weighted $L_1$-norm in the penalized criterion. As for other machine learning methods, cross-validation is very popular for the calibration of the adaptive lasso, that this for the selection of a data-driven optimal value of this hyperparameter. However, the most simple cross-validation scheme is not valid in this context, and a more elaborate one has to be employed to guarantee an optimal calibration. The discrepancy of the simple cross-validation scheme has been well documented in other contexts, but less so when it comes to the calibration of the adaptive lasso, and, therefore, many statistical analysts still overlook it. In this work, we recall appropriate cross-validation schemes for the calibration of the adaptive lasso, and illustrate the discrepancy of the simple scheme, using both synthetic and real-world examples. Our results clearly establish the suboptimality of the simple scheme, in terms of support recovery and prediction error, for several versions of the adaptive lasso, including the popular one-step lasso.