MedVIGIL: Evaluating Trustworthy Medical VLMs Under Broken Visual Evidence

TL;DR

MedVIGIL introduces a comprehensive evaluation suite for medical vision--language models, focusing on their ability to recognize and refuse to answer when visual evidence is compromised, ensuring trustworthy clinical deployment.

Contribution

This work presents a new, clinician-supervised benchmark with 2,556 probes to assess medical VLMs' robustness against evidence failures, including a composite score and public release.

Findings

Radiologist scores show an 83.3 MCS with 5.8% silent failures.

Strongest model achieves 69.2 MCS, leaving room for improvement.

Benchmark and evaluation tools are publicly available.

Abstract

Medical vision--language models (VLMs) are usually evaluated on intact image--question pairs, but trustworthy clinical use requires a stronger property: a model must recognise when the evidential basis for an answer has failed. We study this through silent failures under perturbed evidence, where a vision-required medical question is paired with a false premise, wording perturbation, knowledge-only rewrite, or ROI-corrupted image, yet the model returns a fluent non-refusal answer. We introduce medvigil, a 300-case evaluation suite drawn from four public medical VQA sources, supervised end to end by four board-certified radiologists: every gold answer, refusal option, candidate-answer set, paraphrase, false-premise trap, ROI box, and clinical risk tier is clinician-authored. Two attending radiologists annotate every case in parallel, a senior radiologist consolidates the released manifest, and a separate fourth radiologist independent of construction answers every probe to provide the human reference baseline. The release contains 2{,}556 MCQ probes, 240 counterfactual triplets, physician-adjudicated risk-tier and answerability flags, ROI boxes, and a paired open-ended variant. We report seven correctness-conditioned audit metrics that summarise into the medvigil Composite Score (MCS), and audit 16 vision-capable models plus two text-only baselines. The independent radiologist scores MCS 83.3 at silent-failure rate 5.8%, leaving a 14.1-point composite headroom above the strongest audited model (Claude Opus 4.7 at 69.2). The benchmark and evaluation harness are publicly released.