Feature Importance Ranking for Deep Learning

TL;DR

This paper introduces a dual-net architecture for feature importance ranking in deep learning, effectively addressing combinatorial optimization challenges and outperforming existing methods on various datasets.

Contribution

A novel dual-net framework with joint training and stochastic local search for optimal feature subset discovery and importance ranking in deep learning.

Findings

Outperforms state-of-the-art feature importance methods

Effective on synthetic, benchmark, and real datasets

Addresses combinatorial optimization in feature selection

Abstract

Feature importance ranking has become a powerful tool for explainable AI. However, its nature of combinatorial optimization poses a great challenge for deep learning. In this paper, we propose a novel dual-net architecture consisting of operator and selector for discovery of an optimal feature subset of a fixed size and ranking the importance of those features in the optimal subset simultaneously. During learning, the operator is trained for a supervised learning task via optimal feature subset candidates generated by the selector that learns predicting the learning performance of the operator working on different optimal subset candidates. We develop an alternate learning algorithm that trains two nets jointly and incorporates a stochastic local search procedure into learning to address the combinatorial optimization challenge. In deployment, the selector generates an optimal feature subset and ranks feature importance, while the operator makes predictions based on the optimal subset for test data. A thorough evaluation on synthetic, benchmark and real data sets suggests that our approach outperforms several state-of-the-art feature importance ranking and supervised feature selection methods. (Our source code is available: https://github.com/maksym33/FeatureImportanceDL)