SERPENT-VLM : Self-Refining Radiology Report Generation Using Vision Language Models

TL;DR

SERPENT-VLM introduces a self-refining mechanism in vision-language models to improve radiology report generation, significantly reducing hallucinations and achieving state-of-the-art results on key datasets.

Contribution

It presents a novel self-supervised loss and dynamic interaction framework that enhances image-text alignment in radiology report generation models.

Findings

Outperforms existing models like LLaVA-Med and BiomedGPT.

Achieves state-of-the-art performance on IU X-ray and ROCO datasets.

Demonstrates robustness against noisy images.

Abstract

Radiology Report Generation (R2Gen) demonstrates how Multi-modal Large Language Models (MLLMs) can automate the creation of accurate and coherent radiological reports. Existing methods often hallucinate details in text-based reports that don't accurately reflect the image content. To mitigate this, we introduce a novel strategy, SERPENT-VLM (SElf Refining Radiology RePort GENeraTion using Vision Language Models), which improves the R2Gen task by integrating a self-refining mechanism into the MLLM framework. We employ a unique self-supervised loss that leverages similarity between pooled image representations and the contextual representations of the generated radiological text, alongside the standard Causal Language Modeling objective, to refine image-text representations. This allows the model to scrutinize and align the generated text through dynamic interaction between a given image and the generated text, therefore reducing hallucination and continuously enhancing nuanced report generation. SERPENT-VLM outperforms existing baselines such as LLaVA-Med, BiomedGPT, etc., achieving SoTA performance on the IU X-ray and Radiology Objects in COntext (ROCO) datasets, and also proves to be robust against noisy images. A qualitative case study emphasizes the significant advancements towards more sophisticated MLLM frameworks for R2Gen, opening paths for further research into self-supervised refinement in the medical imaging domain.