Fool Me Once? Contrasting Textual and Visual Explanations in a Clinical Decision-Support Setting

TL;DR

This study evaluates how different explanation modalities in AI-assisted chest X-ray diagnosis influence healthcare practitioners' reliance and decision-making, highlighting the importance of explanation quality and modality combination.

Contribution

It provides the first large-scale user study comparing textual, visual, and combined explanations in a clinical setting, revealing their effects on user reliance and trust.

Findings

Text explanations cause over-reliance, mitigated by saliency maps.

Explanation quality and correctness significantly impact usefulness.

Combining explanations improves user understanding and reduces over-reliance.

Abstract

The growing capabilities of AI models are leading to their wider use, including in safety-critical domains. Explainable AI (XAI) aims to make these models safer to use by making their inference process more transparent. However, current explainability methods are seldom evaluated in the way they are intended to be used: by real-world end users. To address this, we conducted a large-scale user study with 85 healthcare practitioners in the context of human-AI collaborative chest X-ray analysis. We evaluated three types of explanations: visual explanations (saliency maps), natural language explanations, and a combination of both modalities. We specifically examined how different explanation types influence users depending on whether the AI advice and explanations are factually correct. We find that text-based explanations lead to significant over-reliance, which is alleviated by combining them with saliency maps. We also observe that the quality of explanations, that is, how much factually correct information they entail, and how much this aligns with AI correctness, significantly impacts the usefulness of the different explanation types.