Error Controlled Feature Selection for Ultrahigh Dimensional and Highly Correlated Feature Space Using Deep Learning

TL;DR

This paper introduces a deep learning-based feature selection method designed to handle ultra-high dimensional, highly correlated, and noisy data, with controlled error rates, demonstrated through extensive simulations and real data applications.

Contribution

It proposes a novel screening and cleaning approach that adaptively discovers correlated predictors with controlled error, addressing limitations of existing methods in complex feature spaces.

Findings

Achieves high power in feature detection

Maintains low false discovery rate

Effective across diverse simulated and real datasets

Abstract

In recent years, deep learning has been at the center of analytics due to its impressive empirical success in analyzing complex data objects. Despite this success, most of the existing tools behave like black-box machines, thus the increasing interest in interpretable, reliable, and robust deep learning models applicable to a broad class of applications. Feature-selected deep learning has emerged as a promising tool in this realm. However, the recent developments do not accommodate ultra-high dimensional and highly correlated features, in addition to the high noise level. In this article, we propose a novel screening and cleaning method with the aid of deep learning for a data-adaptive multi-resolutional discovery of highly correlated predictors with a controlled error rate. Extensive empirical evaluations over a wide range of simulated scenarios and several real datasets demonstrate the effectiveness of the proposed method in achieving high power while keeping the false discovery rate at a minimum.