Explanation Multiplicity in SHAP: Characterization and Assessment

TL;DR

This paper investigates the phenomenon of explanation multiplicity in SHAP, revealing that multiple valid explanations for the same decision can vary significantly, challenging the reliability of post-hoc interpretability methods.

Contribution

It introduces a comprehensive methodology to characterize explanation multiplicity, disentangles its sources, and provides baseline comparisons to better interpret explanation stability.

Findings

Explanation multiplicity is widespread across datasets and models.

Common stability metrics can mask significant explanation variability.

Baseline comparisons are essential for meaningful interpretation of explanations.

Abstract

Post-hoc explanations are widely used to justify, contest, and review automated decisions in high-stakes domains such as lending, employment, and healthcare. Among these methods, SHAP is often treated as providing a reliable account of which features mattered for an individual prediction and is routinely used to support recourse, oversight, and accountability. In practice, however, SHAP explanations can differ substantially across repeated runs, even when the individual, prediction task, and trained model are held fixed. We conceptualize and name this phenomenon explanation multiplicity: the existence of multiple, internally valid but substantively different explanations for the same decision. Explanation multiplicity poses a normative challenge for responsible AI deployment, as it undermines expectations that explanations can reliably identify the reasons for an adverse outcome. We present a comprehensive methodology for characterizing explanation multiplicity in post-hoc feature attribution methods, disentangling sources arising from model training and selection versus stochasticity intrinsic to the explanation pipeline. Furthermore, whether explanation multiplicity is surfaced depends on how explanation consistency is measured. Commonly used magnitude-based metrics can suggest stability while masking substantial instability in the identity and ordering of top-ranked features. To contextualize observed instability, we derive and estimate randomized baseline values under plausible null models, providing a principled reference point for interpreting explanation disagreement. Across datasets, model classes, and confidence regimes, we find that explanation multiplicity is widespread and persists even under highly controlled conditions, including high-confidence predictions. Thus explanation practices must be evaluated using metrics and baselines aligned with their intended societal role.