Feature importance scores and lossless feature pruning using Banzhaf power indices

TL;DR

This paper introduces a novel method using Banzhaf power indices from coalitional game theory to measure feature influence in classifiers, enabling lossless feature pruning without data access and providing empirical estimates when data is available.

Contribution

It applies Banzhaf power indices to feature importance, allowing lossless feature pruning and offering a data-independent measure of influence in machine learning models.

Findings

Features with zero Banzhaf index can be pruned without accuracy loss

Banzhaf indices can be estimated empirically with data samples

Comparison with gradient saliency and L1 coefficients demonstrates effectiveness

Abstract

Understanding the influence of features in machine learning is crucial to interpreting models and selecting the best features for classification. In this work we propose the use of principles from coalitional game theory to reason about importance of features. In particular, we propose the use of the Banzhaf power index as a measure of influence of features on the outcome of a classifier. We show that features having Banzhaf power index of zero can be losslessly pruned without damage to classifier accuracy. Computing the power indices does not require having access to data samples. However, if samples are available, the indices can be empirically estimated. We compute Banzhaf power indices for a neural network classifier on real-life data, and compare the results with gradient-based feature saliency, and coefficients of a logistic regression model with $L_1$ regularization.