Model-based feature selection for neural networks: A mixed-integer programming approach

TL;DR

This paper introduces a mixed-integer programming method for feature selection in neural networks, enabling significant input reduction while maintaining accuracy, thus improving efficiency and robustness.

Contribution

It presents a novel MILP-based framework for input feature selection in DNNs, specifically tailored for image classification tasks.

Findings

Reduced input size to ~15% of original with maintained accuracy

Produced DNNs with fewer connections, lowering computational cost

Enhanced robustness against adversarial attacks

Abstract

In this work, we develop a novel input feature selection framework for ReLU-based deep neural networks (DNNs), which builds upon a mixed-integer optimization approach. While the method is generally applicable to various classification tasks, we focus on finding input features for image classification for clarity of presentation. The idea is to use a trained DNN, or an ensemble of trained DNNs, to identify the salient input features. The input feature selection is formulated as a sequence of mixed-integer linear programming (MILP) problems that find sets of sparse inputs that maximize the classification confidence of each category. These ''inverse'' problems are regularized by the number of inputs selected for each category and by distribution constraints. Numerical results on the well-known MNIST and FashionMNIST datasets show that the proposed input feature selection allows us to drastically reduce the size of the input to $\sim$15\% while maintaining a good classification accuracy. This allows us to design DNNs with significantly fewer connections, reducing computational effort and producing DNNs that are more robust towards adversarial attacks.