Semi-Analytic Resampling in Lasso

TL;DR

This paper introduces a semi-analytic resampling method for Lasso that reduces computational cost and variance in variable selection, using message passing algorithms and state evolution analysis, validated on simulated and real datasets.

Contribution

It develops a novel semi-analytic resampling approach for Lasso that eliminates repeated sampling, improving efficiency and stability in variable selection tasks.

Findings

Method achieves accurate approximation of resampling averages.

Significant reduction in computational time for bootstrapped Lasso.

Effective variable relevance criterion demonstrated on real-world data.

Abstract

An approximate method for conducting resampling in Lasso, the $\ell_1$ penalized linear regression, in a semi-analytic manner is developed, whereby the average over the resampled datasets is directly computed without repeated numerical sampling, thus enabling an inference free of the statistical fluctuations due to sampling finiteness, as well as a significant reduction of computational time. The proposed method is based on a message passing type algorithm, and its fast convergence is guaranteed by the state evolution analysis, when covariates are provided as zero-mean independently and identically distributed Gaussian random variables. It is employed to implement bootstrapped Lasso (Bolasso) and stability selection, both of which are variable selection methods using resampling in conjunction with Lasso, and resolves their disadvantage regarding computational cost. To examine approximation accuracy and efficiency, numerical experiments were carried out using simulated datasets. Moreover, an application to a real-world dataset, the wine quality dataset, is presented. To process such real-world datasets, an objective criterion for determining the relevance of selected variables is also introduced by the addition of noise variables and resampling.