A Generalized Variable Importance Metric and Estimator for Black Box Machine Learning Models

TL;DR

This paper introduces a new population parameter called GVIM for measuring predictor importance in black box models, providing a causal interpretation and demonstrating its effectiveness through extensive simulations.

Contribution

It defines GVIM for black box models, linking it to CATE for causal interpretation, and validates its estimator with complex simulation studies.

Findings

GVIM can be expressed as a function of CATE.

The proposed estimator performs well in complex simulation scenarios.

GVIM offers a causal perspective on variable importance in black box models.

Abstract

In this paper we define a population parameter, ``Generalized Variable Importance Metric (GVIM)'', to measure importance of predictors for black box machine learning methods, where the importance is not represented by model-based parameter. GVIM is defined for each input variable, using the true conditional expectation function, and it measures the variable's importance in affecting a continuous or a binary response. We extend previously published results to show that the defined GVIM can be represented as a function of the Conditional Average Treatment Effect (CATE) for any kind of a predictor, which gives it a causal interpretation and further justification as an alternative to classical measures of significance that are only available in simple parametric models. Extensive set of simulations using realistically complex relationships between covariates and outcomes and number of regression techniques of varying degree of complexity show the performance of our proposed estimator of the GVIM.