Axiomatic Explainer Globalness via Optimal Transport

TL;DR

This paper introduces Wasserstein Globalness, a new axiomatic complexity measure based on optimal transport, to evaluate and compare the diversity of explanations generated by different explainability methods across various datasets.

Contribution

It proposes a novel axiomatic globalness measure using Wasserstein distance, validated across multiple data types, to enhance explainability method evaluation.

Findings

Wasserstein Globalness effectively differentiates explainers based on explanation diversity.

The measure improves the selection process for explainability methods.

Empirical validation on image, tabular, and synthetic datasets confirms its utility.

Abstract

Explainability methods are often challenging to evaluate and compare. With a multitude of explainers available, practitioners must often compare and select explainers based on quantitative evaluation metrics. One particular differentiator between explainers is the diversity of explanations for a given dataset; i.e. whether all explanations are identical, unique and uniformly distributed, or somewhere between these two extremes. In this work, we define a complexity measure for explainers, globalness, which enables deeper understanding of the distribution of explanations produced by feature attribution and feature selection methods for a given dataset. We establish the axiomatic properties that any such measure should possess and prove that our proposed measure, Wasserstein Globalness, meets these criteria. We validate the utility of Wasserstein Globalness using image, tabular, and synthetic datasets, empirically showing that it both facilitates meaningful comparison between explainers and improves the selection process for explainability methods.