Med-VCD: Mitigating Hallucination for Medical Large Vision Language Models through Visual Contrastive Decoding

TL;DR

Med-VCD is a novel decoding method for medical vision-language models that reduces hallucinations and improves factual accuracy without slowing down inference, by selectively focusing on visually relevant tokens.

Contribution

Introduces Med-VCD, a sparse visual-contrastive decoding approach with token-sparsification that enhances medical LVLM reliability efficiently.

Findings

Raises factual accuracy by 13% on average

Improves hallucination accuracy by 6%

Effective across diverse medical imaging tasks

Abstract

Large vision-language models (LVLMs) are now central to healthcare applications such as medical visual question answering and imaging report generation. Yet, these models remain vulnerable to hallucination outputs that appear plausible but are in fact incorrect. In the natural image domain, several decoding strategies have been proposed to mitigate hallucinations by reinforcing visual evidence, but most rely on secondary decoding or rollback procedures that substantially slow inference. Moreover, existing solutions are often domain-specific and may introduce misalignment between modalities or between generated and ground-truth content. We introduce Med-VCD, a sparse visual-contrastive decoding method that mitigates hallucinations in medical LVLMs without the time overhead of secondary decoding. Med-VCD incorporates a novel token-sparsification strategy that selects visually informed tokens on the fly, trimming redundancy while retaining critical visual context and thus balancing efficiency with reliability. Evaluations on eight medical datasets, spanning ophthalmology, radiology, and pathology tasks in visual question answering, report generation, and dedicated hallucination benchmarks, show that Med-VCD raises factual accuracy by an average of 13\% and improves hallucination accuracy by 6\% relative to baseline medical LVLMs.