Exploring Trust Calibration in XAI - The Impact of Exposing Model Limitations to Lay Users

TL;DR

This study investigates how disclosing model limitations affects trust calibration in explainable AI, revealing that limitation disclosure influences trust judgments but short-term experience does not improve calibration.

Contribution

It provides empirical evidence on the effects of limitation disclosures and stimulus variability on trust calibration in XAI, with publicly available study data and materials.

Findings

Limitation disclosure reliably impacts trust calibration.

Short-term experience does not improve calibration.

Stimulus package variability explains more variance than experimental manipulation.

Abstract

Trust calibration -- aligning user trust judgment with model capability -- is crucial for safe deployment of explainable AI (XAI), yet is often evaluated via global trust ratings detached from objective performance evidence. We present a preregistered, incentivized between-subject online study (N=418 representative UK sample) on explainable skin-lesion classification that disentangles expectation-setting from experienced performance. Participants completed 15 case evaluations using a fixed XAI panel (malignancy score, reliability score, and saliency map). We systematically manipulated five experimental onboarding conditions varying example-based information and limitation disclosures with five stimulus packages naturally varying observed prediction quality. Calibration was operationalized as the deviation between trust-related judgments (TAIS and case-wise ratings) and objective performance benchmarks for the encountered cases, analysed with hierarchical mixed-effects models. Only limitation disclosure for case-wise measures reliably impacts trust calibration, and short-term experience did not yield progressive calibration. Further, the experienced package of stimuli explained substantially more variance than the experimental manipulation. However, participants were hard-pressed to differentiate between case-wise perceived trust, trustworthiness, and accuracy estimation. We discuss implications for designing limitation communication and for measuring and analysing calibration metrics in XAI evaluations. All study materials and data of this study are publicly available for replication and further academic use.