CXR-Agent: Vision-language models for chest X-ray interpretation with uncertainty aware radiology reporting

TL;DR

This paper evaluates state-of-the-art vision-language models for chest X-ray interpretation, identifies hallucination issues, and develops an uncertainty-aware report generation approach that improves accuracy and interpretability.

Contribution

It introduces an agent-based vision-language method for radiology report generation that incorporates uncertainty estimation and localizes pathologies, advancing clinical applicability.

Findings

Vision-language models outperform traditional models on multiple datasets.

Hallucination with confident language is a significant challenge.

Uncertainty-aware reports improve interpretability and safety.

Abstract

Recently large vision-language models have shown potential when interpreting complex images and generating natural language descriptions using advanced reasoning. Medicine's inherently multimodal nature incorporating scans and text-based medical histories to write reports makes it conducive to benefit from these leaps in AI capabilities. We evaluate the publicly available, state of the art, foundational vision-language models for chest X-ray interpretation across several datasets and benchmarks. We use linear probes to evaluate the performance of various components including CheXagent's vision transformer and Q-former, which outperform the industry-standard Torch X-ray Vision models across many different datasets showing robust generalisation capabilities. Importantly, we find that vision-language models often hallucinate with confident language, which slows down clinical interpretation. Based on these findings, we develop an agent-based vision-language approach for report generation using CheXagent's linear probes and BioViL-T's phrase grounding tools to generate uncertainty-aware radiology reports with pathologies localised and described based on their likelihood. We thoroughly evaluate our vision-language agents using NLP metrics, chest X-ray benchmarks and clinical evaluations by developing an evaluation platform to perform a user study with respiratory specialists. Our results show considerable improvements in accuracy, interpretability and safety of the AI-generated reports. We stress the importance of analysing results for normal and abnormal scans separately. Finally, we emphasise the need for larger paired (scan and report) datasets alongside data augmentation to tackle overfitting seen in these large vision-language models.