Semantic Consistency-Based Uncertainty Quantification for Factuality in Radiology Report Generation

TL;DR

This paper introduces a semantic consistency-based method for quantifying uncertainty in radiology report generation, significantly improving factual accuracy and hallucination detection without altering existing models.

Contribution

The proposed plug-and-play framework effectively detects hallucinations and enhances report factuality without requiring model modifications or access to internal states.

Findings

Improves factuality scores by 10% by rejecting 20% of reports.

Achieves 82.9% success rate in flagging lowest-precision sentences.

Effectively detects hallucinations in radiology reports.

Abstract

Radiology report generation (RRG) has shown great potential in assisting radiologists by automating the labor-intensive task of report writing. While recent advancements have improved the quality and coherence of generated reports, ensuring their factual correctness remains a critical challenge. Although generative medical Vision Large Language Models (VLLMs) have been proposed to address this issue, these models are prone to hallucinations and can produce inaccurate diagnostic information. To address these concerns, we introduce a novel Semantic Consistency-Based Uncertainty Quantification framework that provides both report-level and sentence-level uncertainties. Unlike existing approaches, our method does not require modifications to the underlying model or access to its inner state, such as output token logits, thus serving as a plug-and-play module that can be seamlessly integrated with state-of-the-art models. Extensive experiments demonstrate the efficacy of our method in detecting hallucinations and enhancing the factual accuracy of automatically generated radiology reports. By abstaining from high-uncertainty reports, our approach improves factuality scores by $10$\%, achieved by rejecting $20$\% of reports using the \texttt{Radialog} model on the MIMIC-CXR dataset. Furthermore, sentence-level uncertainty flags the lowest-precision sentence in each report with an $82.9$\% success rate. Our implementation is open-source and available at https://github.com/BU-DEPEND-Lab/SCUQ-RRG.