Hallucination Mitigating for Medical Report Generation

TL;DR

This paper presents KERM, a framework that reduces hallucinations in medical report generation by integrating knowledge retrieval, contextual purification, and fine-grained rewards, improving report accuracy and relevance.

Contribution

The work introduces a novel knowledge-enhanced framework with a purification module and reward-guided training to mitigate hallucinations in medical report generation.

Findings

Significant reduction in hallucinations on IU-Xray and MIMIC-CXR datasets.

Enhanced clinical relevance and supportiveness of generated reports.

Improved alignment with medical knowledge and context.

Abstract

In the realm of medical report generation (MRG), the integration of natural language processing has emerged as a vital tool to alleviate the workload of radiologists. Despite the impressive capabilities demonstrated by large vision language models (LVLMs) in understanding natural language, their susceptibility to generating plausible yet inaccurate claims, known as ``hallucinations'', raises concerns-especially in the nuanced and critical field of medical. In this work, we introduce a framework, \textbf{K}nowledge-\textbf{E}nhanced with Fine-Grained \textbf{R}einforced Rewards \textbf{M}edical Report Generation (KERM), to tackle the issue. Our approach refines the input to the LVLM by first utilizing MedCLIP for knowledge retrieval, incorporating relevant lesion fact sentences from a curated knowledge corpus. We then introduce a novel purification module to ensure the retrieved knowledge is contextually relevant to the patient's clinical context. Subsequently, we employ fine-grained rewards to guide these models in generating highly supportive and clinically relevant descriptions, ensuring the alignment of model's outputs with desired behaviors. Experimental results on IU-Xray and MIMIC-CXR datasets validate the effectiveness of our approach in mitigating hallucinations and enhancing report quality.