Human-AI Collaboration in Radiology: The Case of Pulmonary Embolism

TL;DR

This study analyzes how radiologists collaborate with AI in diagnosing pulmonary embolism, revealing high agreement rates, evolving disagreement patterns, workflow improvements, and heterogeneity in AI engagement among radiologists.

Contribution

It provides the first large-scale real-world analysis of radiologist-AI interaction in PE diagnosis, highlighting collaboration dynamics and demographic differences.

Findings

Radiologists agree with AI 84% of positive PE predictions.

Disagreement decreases from 30% to 12% over two years.

AI-assisted workflow increases scan volume without affecting mortality.

Abstract

We study how radiologists use AI to diagnose pulmonary embolism (PE), tracking over 100,000 scans interpreted by nearly 400 radiologists during the staggered rollout of a real-world FDA-approved diagnostic platform in a hospital system. When AI flags PE, radiologists agree 84% of the time; when AI predicts no PE, they agree 97%. Disagreement evolves substantially: radiologists initially reject AI-positive PEs in 30% of cases, dropping to 12% by year two. Despite a 16% increase in scan volume, diagnostic speed remains stable while per-radiologist monthly volumes nearly double, with no change in patient mortality -- suggesting AI improves workflow without compromising outcomes. We document significant heterogeneity in AI collaboration: some radiologists reject AI-flagged PEs half the time while others accept nearly always; female radiologists are 6 percentage points less likely to override AI than male radiologists. Moderate AI engagement is associated with the highest agreement, whereas both low and high engagement show more disagreement. Follow-up imaging reveals that when radiologists override AI to diagnose PE, 54% of subsequent scans show both agreeing on no PE within 30 days.