Do not explain without context: addressing the blind spot of model explanations

TL;DR

This paper examines how the choice of reference data influences the explanations generated by XAI methods, revealing that small distribution changes can significantly alter interpretability and conclusions.

Contribution

It highlights a critical overlooked aspect of model explanations—the impact of reference data—and emphasizes the need for broader context to ensure robustness.

Findings

Small distribution changes can drastically alter explanations

Explanations depend heavily on reference data choice

Robust explanations require broader contextual support

Abstract

The increasing number of regulations and expectations of predictive machine learning models, such as so called right to explanation, has led to a large number of methods promising greater interpretability. High demand has led to a widespread adoption of XAI techniques like Shapley values, Partial Dependence profiles or permutational variable importance. However, we still do not know enough about their properties and how they manifest in the context in which explanations are created by analysts, reviewed by auditors, and interpreted by various stakeholders. This paper highlights a blind spot which, although critical, is often overlooked when monitoring and auditing machine learning models: the effect of the reference data on the explanation calculation. We discuss that many model explanations depend directly or indirectly on the choice of the referenced data distribution. We showcase examples where small changes in the distribution lead to drastic changes in the explanations, such as a change in trend or, alarmingly, a conclusion. Consequently, we postulate that obtaining robust and useful explanations always requires supporting them with a broader context.