Knoop: Practical Enhancement of Knockoff with Over-Parameterization for Variable Selection

TL;DR

Knoop enhances knockoff-based variable selection by generating multiple knockoffs and using over-parameterized models, leading to improved accuracy and robustness in high-dimensional datasets.

Contribution

Introduces Knoop, a novel method that over-parameterizes with multiple knockoffs and anomaly testing for more effective variable selection.

Findings

Higher AUC in ROC for variable identification

Improved predictive accuracy in regression and classification

Robust performance demonstrated in simulations and real data

Abstract

Variable selection plays a crucial role in enhancing modeling effectiveness across diverse fields, addressing the challenges posed by high-dimensional datasets of correlated variables. This work introduces a novel approach namely Knockoff with over-parameterization (Knoop) to enhance Knockoff filters for variable selection. Specifically, Knoop first generates multiple knockoff variables for each original variable and integrates them with the original variables into an over-parameterized Ridgeless regression model. For each original variable, Knoop evaluates the coefficient distribution of its knockoffs and compares these with the original coefficients to conduct an anomaly-based significance test, ensuring robust variable selection. Extensive experiments demonstrate superior performance compared to existing methods in both simulation and real-world datasets. Knoop achieves a notably higher Area under the Curve (AUC) of the Receiver Operating Characteristic (ROC) Curve for effectively identifying relevant variables against the ground truth by controlled simulations, while showcasing enhanced predictive accuracy across diverse regression and classification tasks. The analytical results further backup our observations.