TRIP: A Nonparametric Test to Diagnose Biased Feature Importance Scores

TL;DR

TRIP is a statistical test designed to identify unreliable feature importance scores in machine learning models, especially when permutation methods are misleading due to feature dependencies or model extrapolation.

Contribution

This work introduces TRIP, a minimal-assumption test that detects unreliable permutation feature importance scores, extending its applicability to high-dimensional data.

Findings

TRIP effectively detects unreliable importance scores in simulated data.

The test performs well in real-world applications to identify misleading feature importances.

TRIP can be integrated with existing methods to improve interpretability reliability.

Abstract

Along with accurate prediction, understanding the contribution of each feature to the making of the prediction, i.e., the importance of the feature, is a desirable and arguably necessary component of a machine learning model. For a complex model such as a random forest, such importances are not innate -- as they are, e.g., with linear regression. Efficient methods have been created to provide such capabilities, with one of the most popular among them being permutation feature importance due to its efficiency, model-agnostic nature, and perceived intuitiveness. However, permutation feature importance has been shown to be misleading in the presence of dependent features as a result of the creation of unrealistic observations when permuting the dependent features. In this work, we develop TRIP (Test for Reliable Interpretation via Permutation), a test requiring minimal assumptions that is able to detect unreliable permutation feature importance scores that are the result of model extrapolation. To build on this, we demonstrate how the test can be complemented in order to allow its use in high dimensional settings. Through testing on simulated data and applications, our results show that the test can be used to reliably detect when permutation feature importance scores are unreliable.