Feature relevance quantification in explainable AI: A causal problem

TL;DR

This paper clarifies the correct probabilistic approach for quantifying feature relevance in explainable AI, emphasizing the importance of using unconditional expectations over conditional ones, and critiques current methods like SHAP.

Contribution

It provides a conceptual clarification based on causality theory, distinguishing between observational and interventional probabilities, and critiques existing implementations of SHAP.

Findings

Unconditional expectations are the correct basis for feature dropping.

Current SHAP implementations approximate conditional expectations, which may be flawed.

The distinction impacts the interpretation of feature relevance in explainable AI.

Abstract

We discuss promising recent contributions on quantifying feature relevance using Shapley values, where we observed some confusion on which probability distribution is the right one for dropped features. We argue that the confusion is based on not carefully distinguishing between observational and interventional conditional probabilities and try a clarification based on Pearl's seminal work on causality. We conclude that unconditional rather than conditional expectations provide the right notion of dropping features in contradiction to the theoretical justification of the software package SHAP. Parts of SHAP are unaffected because unconditional expectations (which we argue to be conceptually right) are used as approximation for the conditional ones, which encouraged others to `improve' SHAP in a way that we believe to be flawed.