Localizing Before Answering: A Hallucination Evaluation Benchmark for Grounded Medical Multimodal LLMs

TL;DR

This paper introduces HEAL-MedVQA, a benchmark for evaluating localization and hallucination in medical multimodal models, and proposes the LobA framework to improve their grounded reasoning and answer reliability.

Contribution

It presents a new benchmark with evaluation protocols and a large dataset for localization assessment, along with a training framework that enhances model grounding and reduces hallucinations.

Findings

LobA significantly improves localization accuracy.

Models trained with LobA produce more reliable, grounded answers.

HEAL-MedVQA effectively evaluates hallucination robustness in medical LMMs.

Abstract

Medical Large Multi-modal Models (LMMs) have demonstrated remarkable capabilities in medical data interpretation. However, these models frequently generate hallucinations contradicting source evidence, particularly due to inadequate localization reasoning. This work reveals a critical limitation in current medical LMMs: instead of analyzing relevant pathological regions, they often rely on linguistic patterns or attend to irrelevant image areas when responding to disease-related queries. To address this, we introduce HEAL-MedVQA (Hallucination Evaluation via Localization MedVQA), a comprehensive benchmark designed to evaluate LMMs' localization abilities and hallucination robustness. HEAL-MedVQA features (i) two innovative evaluation protocols to assess visual and textual shortcut learning, and (ii) a dataset of 67K VQA pairs, with doctor-annotated anatomical segmentation masks for pathological regions. To improve visual reasoning, we propose the Localize-before-Answer (LobA) framework, which trains LMMs to localize target regions of interest and self-prompt to emphasize segmented pathological areas, generating grounded and reliable answers. Experimental results demonstrate that our approach significantly outperforms state-of-the-art biomedical LMMs on the challenging HEAL-MedVQA benchmark, advancing robustness in medical VQA.