When Features Beat Noise: A Feature Selection Technique Through Noise-Based Hypothesis Testing

TL;DR

This paper introduces a statistically grounded feature selection method that uses noise-based hypothesis testing, outperforming existing techniques in both simulated and real-world datasets by reliably identifying informative predictors.

Contribution

The paper presents a novel feature selection approach that incorporates multiple noise features and bootstrap hypothesis testing, providing a theoretical foundation and improved performance over existing methods.

Findings

Consistently outperforms Boruta and Knockoff in recovering signals.

Demonstrates robustness across diverse real-world datasets.

Offers a principled, statistically justified feature selection framework.

Abstract

Feature selection has remained a daunting challenge in machine learning and artificial intelligence, where increasingly complex, high-dimensional datasets demand principled strategies for isolating the most informative predictors. Despite widespread adoption, many established techniques suffer from notable limitations; some incur substantial computational cost, while others offer no definite statistical driven stopping criteria or assesses the significance of their importance scores. A common heuristic approach introduces multiple random noise features and retains all predictors ranked above the strongest noise feature. Although intuitive, this strategy lacks theoretical justification and depends heavily on heuristics. This paper proposes a novel feature selection method that addresses these limitations. Our approach introduces multiple random noise features and evaluates each feature's importance against the maximum importance value among these noise features incorporating a non-parametric bootstrap-based hypothesis testing framework to establish a solid theoretical foundation. We establish the conceptual soundness of our approach through statistical derivations that articulate the principles guiding the design of our algorithm. To evaluate its reliability, we generated simulated datasets under controlled statistical settings and benchmarked performance against Boruta and Knockoff-based methods, observing consistently stronger recovery of meaningful signal. As a demonstration of practical utility, we applied the technique across diverse real-world datasets, where it surpassed feature selection techniques including Boruta, RFE, and Extra Trees. Hence, the method emerges as a robust algorithm for principled feature selection, enabling the distillation of informative predictors that support reliable inference, enhanced predictive performance, and efficient computation.