Consensus, dissensus and synergy between clinicians and specialist foundation models in radiology report generation

TL;DR

This study develops a state-of-the-art AI system for radiology report generation, demonstrating that AI can produce reports preferred or deemed equivalent to human reports in clinical settings, and explores clinician-AI collaboration.

Contribution

The paper introduces Flamingo-CXR, a novel vision-language model fine-tuned for radiology report generation and evaluates its clinical quality through radiologist assessments, highlighting AI-human complementarity.

Findings

AI reports preferred in over 60% of cases by radiologists.

AI-generated reports often contain location and finding errors.

Clinician-AI collaboration improves report quality, with 80% of inpatient reports deemed equivalent or better.

Abstract

Radiology reports are an instrumental part of modern medicine, informing key clinical decisions such as diagnosis and treatment. The worldwide shortage of radiologists, however, restricts access to expert care and imposes heavy workloads, contributing to avoidable errors and delays in report delivery. While recent progress in automated report generation with vision-language models offer clear potential in ameliorating the situation, the path to real-world adoption has been stymied by the challenge of evaluating the clinical quality of AI-generated reports. In this study, we build a state-of-the-art report generation system for chest radiographs, $\textit{Flamingo-CXR}$, by fine-tuning a well-known vision-language foundation model on radiology data. To evaluate the quality of the AI-generated reports, a group of 16 certified radiologists provide detailed evaluations of AI-generated and human written reports for chest X-rays from an intensive care setting in the United States and an inpatient setting in India. At least one radiologist (out of two per case) preferred the AI report to the ground truth report in over 60$\%$ of cases for both datasets. Amongst the subset of AI-generated reports that contain errors, the most frequently cited reasons were related to the location and finding, whereas for human written reports, most mistakes were related to severity and finding. This disparity suggested potential complementarity between our AI system and human experts, prompting us to develop an assistive scenario in which Flamingo-CXR generates a first-draft report, which is subsequently revised by a clinician. This is the first demonstration of clinician-AI collaboration for report writing, and the resultant reports are assessed to be equivalent or preferred by at least one radiologist to reports written by experts alone in 80$\%$ of in-patient cases and 60$\%$ of intensive care cases.