Augmenting Clinical Decision-Making with an Interactive and Interpretable AI Copilot: A Real-World User Study with Clinicians in Nephrology and Obstetrics

TL;DR

This study introduces AICare, an interactive and interpretable AI tool designed to support clinicians in nephrology and obstetrics, demonstrating reduced workload and varied trust-building strategies through a real-world user study.

Contribution

It presents AICare, a novel AI copilot with visual explanations and diagnostic suggestions, evaluated in clinical settings to enhance decision-making and trust.

Findings

AICare reduces clinicians' cognitive workload.

Clinicians use AICare differently based on expertise.

Trust is built through verification and interaction strategies.

Abstract

Clinician skepticism toward opaque AI hinders adoption in high-stakes healthcare. We present AICare, an interactive and interpretable AI copilot for collaborative clinical decision-making. By analyzing longitudinal electronic health records, AICare grounds dynamic risk predictions in scrutable visualizations and LLM-driven diagnostic recommendations. Through a within-subjects counterbalanced study with 16 clinicians across nephrology and obstetrics, we comprehensively evaluated AICare using objective measures (task completion time and error rate), subjective assessments (NASA-TLX, SUS, and confidence ratings), and semi-structured interviews. Our findings indicate AICare's reduced cognitive workload. Beyond performance metrics, qualitative analysis reveals that trust is actively constructed through verification, with interaction strategies diverging by expertise: junior clinicians used the system as cognitive scaffolding to structure their analysis, while experts engaged in adversarial verification to challenge the AI's logic. This work offers design implications for creating AI systems that function as transparent partners, accommodating diverse reasoning styles to augment rather than replace clinical judgment.