Dynamic Instance-Wise Classification in Correlated Feature Spaces

TL;DR

This paper introduces a Bayesian network-based method for dynamic, instance-wise feature selection in correlated feature spaces, improving classification accuracy and interpretability by tailoring features to each test instance.

Contribution

It proposes a novel sequential feature selection approach that adapts to each test instance using feature dependencies, with theoretical analysis and a scalable algorithm for high-dimensional data.

Findings

Improved classification accuracy on real-world datasets.

Enhanced interpretability through instance-specific feature subsets.

Scalable method effective in high-dimensional settings.

Abstract

In a typical supervised machine learning setting, the predictions on all test instances are based on a common subset of features discovered during model training. However, using a different subset of features that is most informative for each test instance individually may not only improve prediction accuracy, but also the overall interpretability of the model. At the same time, feature selection methods for classification have been known to be the most effective when many features are irrelevant and/or uncorrelated. In fact, feature selection ignoring correlations between features can lead to poor classification performance. In this work, a Bayesian network is utilized to model feature dependencies. Using the dependency network, a new method is proposed that sequentially selects the best feature to evaluate for each test instance individually, and stops the selection process to make a prediction once it determines that no further improvement can be achieved with respect to classification accuracy. The optimum number of features to acquire and the optimum classification strategy are derived for each test instance. The theoretical properties of the optimum solution are analyzed, and a new algorithm is proposed that takes advantage of these properties to implement a robust and scalable solution for high dimensional settings. The effectiveness, generalizability, and scalability of the proposed method is illustrated on a variety of real-world datasets from diverse application domains.