Knowing or Guessing? Robust Medical Visual Question Answering via Joint Consistency and Contrastive Learning

TL;DR

This paper identifies robustness issues in medical visual question answering models due to semantic misalignment and biases, and proposes a joint consistency and contrastive learning approach to improve reliability and answer stability.

Contribution

It introduces RoMed, a new dataset for evaluating robustness, and proposes CCL, a novel training method combining consistency and contrastive learning to enhance model robustness.

Findings

Significant performance drops of SOTA models on RoMed dataset.

CCL improves answer consistency by 50% on RoMed.

State-of-the-art results achieved on multiple VQA benchmarks.

Abstract

In high-stakes medical applications, consistent answering across diverse question phrasings is essential for reliable diagnosis. However, we reveal that current Medical Vision-Language Models (Med-VLMs) exhibit concerning fragility in Medical Visual Question Answering, as their answers fluctuate significantly when faced with semantically equivalent rephrasings of medical questions. We attribute this to two limitations: (1) insufficient alignment of medical concepts, leading to divergent reasoning patterns, and (2) hidden biases in training data that prioritize syntactic shortcuts over semantic understanding. To address these challenges, we construct RoMed, a dataset built upon original VQA datasets containing 144k questions with variations spanning word-level, sentence-level, and semantic-level perturbations. When evaluating state-of-the-art (SOTA) models like LLaVA-Med on RoMed, we observe alarming performance drops (e.g., a 40\% decline in Recall) compared to original VQA benchmarks, exposing critical robustness gaps. To bridge this gap, we propose Consistency and Contrastive Learning (CCL), which integrates two key components: (1) knowledge-anchored consistency learning, aligning Med-VLMs with medical knowledge rather than shallow feature patterns, and (2) bias-aware contrastive learning, mitigating data-specific priors through discriminative representation refinement. CCL achieves SOTA performance on three popular VQA benchmarks and notably improves answer consistency by 50\% on the challenging RoMed test set, demonstrating significantly enhanced robustness. Code will be released.