ET-Lasso: A New Efficient Tuning of Lasso-type Regularization for High-Dimensional Data

TL;DR

ET-Lasso introduces a novel, efficient method for tuning Lasso regularization by using permuted pseudo-features to improve feature selection accuracy in high-dimensional data.

Contribution

The paper proposes ET-Lasso, a new tuning approach that leverages pseudo-features to better distinguish active from inactive features, enhancing feature selection in high-dimensional settings.

Findings

ET-Lasso effectively separates active and inactive features.

It outperforms traditional tuning methods in accuracy and efficiency.

Demonstrated success on both simulated and real-world datasets.

Abstract

The L1 regularization (Lasso) has proven to be a versatile tool to select relevant features and estimate the model coefficients simultaneously and has been widely used in many research areas such as genomes studies, finance, and biomedical imaging. Despite its popularity, it is very challenging to guarantee the feature selection consistency of Lasso especially when the dimension of the data is huge. One way to improve the feature selection consistency is to select an ideal tuning parameter. Traditional tuning criteria mainly focus on minimizing the estimated prediction error or maximizing the posterior model probability, such as cross-validation and BIC, which may either be time-consuming or fail to control the false discovery rate (FDR) when the number of features is extremely large. The other way is to introduce pseudo-features to learn the importance of the original ones. Recently, the Knockoff filter is proposed to control the FDR when performing feature selection. However, its performance is sensitive to the choice of the expected FDR threshold. Motivated by these ideas, we propose a new method using pseudo-features to obtain an ideal tuning parameter. In particular, we present the Efficient Tuning of Lasso (ET-Lasso) to separate active and inactive features by adding permuted features as pseudo-features in linear models. The pseudo-features are constructed to be inactive by nature, which can be used to obtain a cutoff to select the tuning parameter that separates active and inactive features. Experimental studies on both simulations and real-world data applications are provided to show that ET-Lasso can effectively and efficiently select active features under a wide range of scenarios