Rethinking XAI Evaluation: A Human-Centered Audit of Shapley Benchmarks in High-Stakes Settings

TL;DR

This paper critically examines how current Shapley value explanations for AI are evaluated, revealing a disconnect between quantitative metrics and human decision-making in high-stakes environments.

Contribution

It introduces a unified framework to compare Shapley variants and demonstrates that existing metrics do not align with human utility or decision confidence.

Findings

Quantitative metrics are decoupled from human-perceived clarity.

Explanations increase decision confidence but do not improve objective performance.

Current evaluation proxies are insufficient for predicting human impact.

Abstract

Shapley values are a cornerstone of explainable AI, yet their proliferation into competing formulations has created a fragmented landscape with little consensus on practical deployment. While theoretical differences are well-documented, evaluation remains reliant on quantitative proxies whose alignment with human utility is unverified. In this work, we use a unified amortized framework to isolate semantic differences between eight Shapley variants under the low-latency constraints of operational risk workflows. We conduct a large-scale empirical evaluation across four risk datasets and a realistic fraud-detection environment involving professional analysts and 3,735 case reviews. Our results reveal a fundamental misalignment: standard quantitative metrics, such as sparsity and faithfulness, are decoupled from human-perceived clarity and decision utility. Furthermore, while no formulation improved objective analyst performance, explanations consistently increased decision confidence, signaling a critical risk of automation bias in high-stakes settings. These findings suggest that current evaluation proxies are insufficient for predicting downstream human impact, and we provide evidence-based guidance for selecting formulations and metrics in operational decision systems.