An exploration of the influence of path choice in game-theoretic attribution algorithms

TL;DR

This paper compares path-dependent feature attribution methods in game-theoretic explanations, showing how path choice impacts attribution consistency and advocating for straight-line paths for robustness and data manifold proximity.

Contribution

It provides a theoretical analysis of how path choice affects attribution methods like interventional Shapley and GIG, highlighting the advantages of straight-line paths.

Findings

Interventional Shapley is equivalent to multi-path integration over all feature permutations.

Straight-line paths yield more consistent attributions than multi-path approaches.

GIG remains robust for decision trees and atomic games.

Abstract

We compare machine learning explainability methods based on the theory of atomic (Shapley, 1953) and infinitesimal (Aumann and Shapley, 1974) games, in a theoretical and experimental investigation into how the model and choice of integration path can influence the resulting feature attributions. To gain insight into differences in attributions resulting from interventional Shapley values (Sundararajan and Najmi, 2019; Janzing et al., 2019; Chen et al., 2019) and Generalized Integrated Gradients (GIG) (Merrill et al., 2019) we note interventional Shapley is equivalent to a multi-path integration along $n!$ paths where $n$ is the number of model input features. Applying Stoke's theorem we show that the path symmetry of these two methods results in the same attributions when the model is composed of a sum of separable functions of individual features and a sum of two-feature products. We then perform a series of experiments with varying degrees of data missingness to demonstrate how interventional Shapley's multi-path approach can yield less consistent attributions than the single straight-line path of Aumann-Shapley. We argue this is because the multiple paths employed by interventional Shapley extend away from the training data manifold and are therefore more likely to pass through regions where the model has little support. In the absence of a more meaningful path choice, we therefore advocate the straight-line path since it will almost always pass closer to the data manifold. Among straight-line path attribution algorithms, GIG is uniquely robust since it will still yield Shapley values for atomic games modeled by decision trees.