Closing the Performance Gap Between AI and Radiologists in Chest X-Ray Reporting

TL;DR

This paper introduces MAIRA-X, a multimodal AI model for chest X-ray report generation that improves report quality and assists radiologists, validated on large datasets and with a user study showing comparable error rates to human reports.

Contribution

The paper presents MAIRA-X, a novel multimodal AI system for comprehensive chest X-ray reporting, including pathological findings and lines/tubes, trained on a large-scale dataset and evaluated with a first-of-its-kind user study.

Findings

MAIRA-X outperforms state-of-the-art models on multiple datasets.

User study shows AI reports have similar error rates to human reports.

The model effectively assists radiologists in high-volume clinical settings.

Abstract

AI-assisted report generation offers the opportunity to reduce radiologists' workload stemming from expanded screening guidelines, complex cases and workforce shortages, while maintaining diagnostic accuracy. In addition to describing pathological findings in chest X-ray reports, interpreting lines and tubes (L&T) is demanding and repetitive for radiologists, especially with high patient volumes. We introduce MAIRA-X, a clinically evaluated multimodal AI model for longitudinal chest X-ray (CXR) report generation, that encompasses both clinical findings and L&T reporting. Developed using a large-scale, multi-site, longitudinal dataset of 3.1 million studies (comprising 6 million images from 806k patients) from Mayo Clinic, MAIRA-X was evaluated on three holdout datasets and the public MIMIC-CXR dataset, where it significantly improved AI-generated reports over the state of the art on lexical quality, clinical correctness, and L&T-related elements. A novel L&T-specific metrics framework was developed to assess accuracy in reporting attributes such as type, longitudinal change and placement. A first-of-its-kind retrospective user evaluation study was conducted with nine radiologists of varying experience, who blindly reviewed 600 studies from distinct subjects. The user study found comparable rates of critical errors (3.0% for original vs. 4.6% for AI-generated reports) and a similar rate of acceptable sentences (97.8% for original vs. 97.4% for AI-generated reports), marking a significant improvement over prior user studies with larger gaps and higher error rates. Our results suggest that MAIRA-X can effectively assist radiologists, particularly in high-volume clinical settings.