A Nonparametric Statistics Approach to Feature Selection in Deep Neural Networks with Theoretical Guarantees

TL;DR

This paper introduces a gradient-descent-free feature selection method for deep neural networks that guarantees consistency and performs well in high-dimensional, nonlinear settings with complex feature interactions.

Contribution

It reformulates neural networks as index models and uses Stein's formula for feature selection, providing theoretical guarantees without relying on traditional optimization.

Findings

Guarantees feature selection consistency with n = Ω(p^2)

Achieves nonlinear selection consistency with n = Ω(s log p)

Demonstrates strong empirical performance on simulations and real data

Abstract

This paper tackles the problem of feature selection in a highly challenging setting: $\mathbb{E}(y | \boldsymbol{x}) = G(\boldsymbol{x}_{\mathcal{S}_0})$, where $\mathcal{S}_0$ is the set of relevant features and $G$ is an unknown, potentially nonlinear function subject to mild smoothness conditions. Our approach begins with feature selection in deep neural networks, then generalizes the results to H{\"o}lder smooth functions by exploiting the strong approximation capabilities of neural networks. Unlike conventional optimization-based deep learning methods, we reformulate neural networks as index models and estimate $\mathcal{S}_0$ using the second-order Stein's formula. This gradient-descent-free strategy guarantees feature selection consistency with a sample size requirement of $n = \Omega(p^2)$, where $p$ is the feature dimension. To handle high-dimensional scenarios, we further introduce a screening-and-selection mechanism that achieves nonlinear selection consistency when $n = \Omega(s \log p)$, with $s$ representing the sparsity level. Additionally, we refit a neural network on the selected features for prediction and establish performance guarantees under a relaxed sparsity assumption. Extensive simulations and real-data analyses demonstrate the strong performance of our method even in the presence of complex feature interactions.