Does Explanation Correctness Matter? Linking Computational XAI Evaluation to Human Understanding

TL;DR

This study experimentally tests how varying levels of explanation correctness in XAI affect human understanding, revealing that lower correctness reduces learning proportion but not always accuracy, highlighting the need for human-centered validation.

Contribution

It provides empirical evidence linking explanation correctness levels to human understanding, challenging assumptions about the direct correlation between correctness metrics and user comprehension.

Findings

Performance drops at 70% and 55% correctness levels.

Lower correctness decreases the proportion of participants who learn the decision pattern.

Fully correct explanations do not guarantee understanding for all participants.

Abstract

Explainable AI (XAI) methods are commonly evaluated with functional metrics such as correctness, which computationally estimate how accurately an explanation reflects the model's reasoning. Higher correctness is assumed to produce better human understanding, but this link has not been tested experimentally with controlled levels. We conducted a user study (N=200) that manipulated explanation correctness at four levels (100%, 85%, 70%, 55%) in a time series classification task where participants could not rely on domain knowledge or visual intuition and instead predicted the AI's decisions based on explanations (forward simulation). Correctness affected understanding, but not at every level: performance dropped at 70% and 55% correctness relative to fully correct explanations, while further degradation below 70% produced no additional loss. Rather than shifting performance uniformly, lower correctness decreased the proportion of participants who learned the decision pattern. At the same time, even fully correct explanations did not guarantee understanding, as only a subset of participants achieved high accuracy. Exploratory analyses showed that self-reported ratings correlated with demonstrated performance only when explanations were fully correct and participants had learned the pattern. These findings show that not all differences in functional correctness translate to differences in human understanding, underscoring the need to validate functional metrics against human outcomes.