Evaluating the Ability of Explanations to Disambiguate Models in a Rashomon Set

TL;DR

This paper introduces AXE, a new method for evaluating feature-importance explanations in AI models, which effectively detects false explanations and helps select models within a Rashomon set that are both accurate and interpretable.

Contribution

The paper proposes AXE, a novel explanation evaluation method based on three principles, capable of detecting adversarial explanations and addressing limitations of prior evaluation strategies.

Findings

AXE detects false explanations with 100% success rate.

Evaluation metrics based on ground truth can obscure behavioral differences.

AXE can identify when protected attributes influence model predictions.

Abstract

Explainable artificial intelligence (XAI) is concerned with producing explanations indicating the inner workings of models. For a Rashomon set of similarly performing models, explanations provide a way of disambiguating the behavior of individual models, helping select models for deployment. However explanations themselves can vary depending on the explainer used, and need to be evaluated. In the paper "Evaluating Model Explanations without Ground Truth", we proposed three principles of explanation evaluation and a new method "AXE" to evaluate the quality of feature-importance explanations. We go on to illustrate how evaluation metrics that rely on comparing model explanations against ideal ground truth explanations obscure behavioral differences within a Rashomon set. Explanation evaluation aligned with our proposed principles would highlight these differences instead, helping select models from the Rashomon set. The selection of alternate models from the Rashomon set can maintain identical predictions but mislead explainers into generating false explanations, and mislead evaluation methods into considering the false explanations to be of high quality. AXE, our proposed explanation evaluation method, can detect this adversarial fairwashing of explanations with a 100% success rate. Unlike prior explanation evaluation strategies such as those based on model sensitivity or ground truth comparison, AXE can determine when protected attributes are used to make predictions.