Consistent Feature Selection for Analytic Deep Neural Networks

TL;DR

This paper establishes the theoretical consistency of feature selection methods, specifically Adaptive Group Lasso, for various deep neural network architectures, addressing a key gap in interpretability research.

Contribution

It proves that Adaptive Group Lasso achieves feature selection consistency for a broad class of deep neural networks, filling a significant theoretical gap.

Findings

Adaptive Group Lasso is consistent for deep neural networks

Group Lasso may be inefficient for neural network feature selection

Theoretical foundation supports interpretability of deep models

Abstract

One of the most important steps toward interpretability and explainability of neural network models is feature selection, which aims to identify the subset of relevant features. Theoretical results in the field have mostly focused on the prediction aspect of the problem with virtually no work on feature selection consistency for deep neural networks due to the model's severe nonlinearity and unidentifiability. This lack of theoretical foundation casts doubt on the applicability of deep learning to contexts where correct interpretations of the features play a central role. In this work, we investigate the problem of feature selection for analytic deep networks. We prove that for a wide class of networks, including deep feed-forward neural networks, convolutional neural networks, and a major sub-class of residual neural networks, the Adaptive Group Lasso selection procedure with Group Lasso as the base estimator is selection-consistent. The work provides further evidence that Group Lasso might be inefficient for feature selection with neural networks and advocates the use of Adaptive Group Lasso over the popular Group Lasso.