When Medical AI Explanations Help and When They Harm

TL;DR

This study reveals that AI explanations in medical diagnosis improve accuracy when correct but harm decision quality when AI errs, due to physicians over-relying on explanations regardless of their correctness.

Contribution

It uncovers the paradoxical effect of AI explanations in medicine, showing they can both help and harm depending on AI accuracy, and quantifies the welfare implications of transparency policies.

Findings

Explanations increase accuracy by 6.3 percentage points when AI is correct.

Explanations decrease accuracy by 4.9 points when AI is incorrect.

Selective transparency yields more healthcare value than universal transparency.

Abstract

We document a fundamental paradox in AI transparency: explanations improve decisions when algorithms are correct but systematically worsen them when algorithms err. In an experiment with 257 medical students making 3,855 diagnostic decisions, we find explanations increase accuracy by 6.3 percentage points when AI is correct (73% of cases) but decrease it by 4.9 points when incorrect (27% of cases). This asymmetry arises because modern AI systems generate equally persuasive explanations regardless of recommendation quality-physicians cannot distinguish helpful from misleading guidance. We show physicians treat explained AI as 15.2 percentage points more accurate than reality, with over-reliance persisting even for erroneous recommendations. Competent physicians with appropriate uncertainty suffer most from the AI transparency paradox (-12.4pp when AI errs), while overconfident novices benefit most (+9.9pp net). Welfare analysis reveals that selective transparency generates \$2.59 billion in annual healthcare value, 43% more than the \$1.82 billion from mandated universal transparency.