How to Measure Human-AI Prediction Accuracy in Explainable AI Systems

TL;DR

This paper proposes new mathematical methods to measure partial wrongness in human predictions of AI actions, enabling more nuanced evaluation of explainable AI systems especially in large output spaces.

Contribution

It introduces three mathematical bases for quantifying partial wrongness and demonstrates their application through two decision-making domain analyses.

Findings

Effective measurement of partial wrongness in large output spaces.

Improved rigor in user studies of explainable AI predictions.

Validated methods through in-lab and re-analysis studies.

Abstract

Assessing an AI system's behavior-particularly in Explainable AI Systems-is sometimes done empirically, by measuring people's abilities to predict the agent's next move-but how to perform such measurements? In empirical studies with humans, an obvious approach is to frame the task as binary (i.e., prediction is either right or wrong), but this does not scale. As output spaces increase, so do floor effects, because the ratio of right answers to wrong answers quickly becomes very small. The crux of the problem is that the binary framing is failing to capture the nuances of the different degrees of "wrongness." To address this, we begin by proposing three mathematical bases upon which to measure "partial wrongness." We then uses these bases to perform two analyses on sequential decision-making domains: the first is an in-lab study with 86 participants on a size-36 action space; the second is a re-analysis of a prior study on a size-4 action space. Other researchers adopting our operationalization of the prediction task and analysis methodology will improve the rigor of user studies conducted with that task, which is particularly important when the domain features a large output space.