Bridging Vision and Language: Optimal Transport-Driven Radiology Report Generation via LLMs

TL;DR

This paper introduces OTDRG, a novel framework that uses Optimal Transport to align X-ray image features with disease labels, enhancing the clinical accuracy and linguistic quality of radiology reports generated by LLMs.

Contribution

The paper proposes a new OT-based alignment method and a disease prediction module to improve the clinical relevance of LLM-generated radiology reports.

Findings

Achieves state-of-the-art results on MIMIC-CXR and IU X-Ray datasets.

Reports are both linguistically coherent and clinically accurate.

Outperforms existing methods in natural language generation and clinical efficacy metrics.

Abstract

Radiology report generation represents a significant application within medical AI, and has achieved impressive results. Concurrently, large language models (LLMs) have demonstrated remarkable performance across various domains. However, empirical validation indicates that general LLMs tend to focus more on linguistic fluency rather than clinical effectiveness, and lack the ability to effectively capture the relationship between X-ray images and their corresponding texts, thus resulting in poor clinical practicability. To address these challenges, we propose Optimal Transport-Driven Radiology Report Generation (OTDRG), a novel framework that leverages Optimal Transport (OT) to align image features with disease labels extracted from reports, effectively bridging the cross-modal gap. The core component of OTDRG is Alignment \& Fine-Tuning, where OT utilizes results from the encoding of label features and image visual features to minimize cross-modal distances, then integrating image and text features for LLMs fine-tuning. Additionally, we design a novel disease prediction module to predict disease labels contained in X-ray images during validation and testing. Evaluated on the MIMIC-CXR and IU X-Ray datasets, OTDRG achieves state-of-the-art performance in both natural language generation (NLG) and clinical efficacy (CE) metrics, delivering reports that are not only linguistically coherent but also clinically accurate.