To Agree or To Be Right? The Grounding-Sycophancy Tradeoff in Medical Vision-Language Models

TL;DR

This study evaluates medical vision-language models revealing a tradeoff between grounding accuracy and sycophantic tendencies, highlighting the need for balanced robustness for clinical deployment.

Contribution

It introduces three novel metrics to quantify grounding and sycophancy, and demonstrates that current models cannot simultaneously excel in both, emphasizing the importance of joint evaluation.

Findings

Models with low hallucination are highly sycophantic.

Most models exhibit poor combined grounding and safety scores.

No model exceeds a Clinical Safety Index of 0.35.

Abstract

Vision-language models (VLMs) adapted to the medical domain have shown strong performance on visual question answering benchmarks, yet their robustness against two critical failure modes, hallucination and sycophancy, remains poorly understood, particularly in combination. We evaluate six VLMs (three general-purpose, three medical-specialist) on three medical VQA datasets and uncover a grounding-sycophancy tradeoff: models with the lowest hallucination propensity are the most sycophantic, while the most pressure-resistant model hallucinates more than all medical-specialist models. To characterize this tradeoff, we propose three metrics: L-VASE, a logit-space reformulation of VASE that avoids its double-normalization; CCS, a confidence-calibrated sycophancy score that penalizes high-confidence capitulation; and Clinical Safety Index (CSI), a unified safety index that combines grounding, autonomy, and calibration via a geometric mean. Across 1,151 test cases, no model achieves a CSI above 0.35, indicating that none of the evaluated 7-8B parameter VLMs is simultaneously well-grounded and robust to social pressure. Our findings suggest that joint evaluation of both properties is necessary before these models can be considered for clinical use. Code is available at https://github.com/UTSA-VIRLab/AgreeOrRight