Stability Selection via Variable Decorrelation

TL;DR

This paper introduces a simple decorrelation step before applying Lasso to improve variable selection stability, effective in both high- and low-dimensional data, with empirical validation and a supporting R package.

Contribution

The paper proposes a variable decorrelation method that enhances Lasso stability and satisfies the irrepresentable condition, applicable across different data dimensions and variable selection techniques.

Findings

Decorrelating variables improves Lasso stability.

The method satisfies the irrepresentable condition after decorrelation.

Empirical results demonstrate effectiveness across techniques.

Abstract

The Lasso is a prominent algorithm for variable selection. However, its instability in the presence of correlated variables in the high-dimensional setting is well-documented. Although previous research has attempted to address this issue by modifying the Lasso loss function, this paper introduces an approach that simplifies the data processed by Lasso. We propose that decorrelating variables before applying the Lasso improves the stability of variable selection regardless of the direction of correlation among predictors. Furthermore, we highlight that the irrepresentable condition, which ensures consistency for the Lasso, is satisfied after variable decorrelation under two assumptions. In addition, by noting that the instability of the Lasso is not limited to high-dimensional settings, we demonstrate the effectiveness of the proposed approach for low-dimensional data. Finally, we present empirical results that indicate the efficacy of the proposed method across different variable selection techniques, highlighting its potential for broader application. The DVS R package is developed to facilitate the implementation of the methodology proposed in this paper.