Benchmarking Deflection and Hallucination in Large Vision-Language Models

TL;DR

This paper introduces VLM-DeflectionBench, a new benchmark and evaluation protocol to assess how large vision-language models handle conflicting or incomplete retrieval evidence, emphasizing deflection behavior.

Contribution

It presents a dynamic data curation pipeline, a comprehensive benchmark dataset, and a fine-grained evaluation protocol to improve understanding of model responses under retrieval challenges.

Findings

Models often fail to deflect with noisy or misleading evidence.

Evaluation reveals models' behavior when knowledge is incomplete or conflicting.

Benchmark enables assessment of retrieval robustness versus parametric memorization.

Abstract

Large Vision-Language Models (LVLMs) increasingly rely on retrieval to answer knowledge-intensive multimodal questions. Existing benchmarks overlook conflicts between visual and textual evidence and the importance of generating deflections (e.g., Sorry, I cannot answer...) when retrieved knowledge is incomplete. These benchmarks also suffer from rapid obsolescence, as growing LVLM training sets allow models to answer many questions without retrieval. We address these gaps with three contributions. First, we propose a dynamic data curation pipeline that preserves benchmark difficulty over time by filtering for genuinely retrieval-dependent samples. Second, we introduce VLM-DeflectionBench, a benchmark of 2,775 samples spanning diverse multimodal retrieval settings, designed to probe model behaviour under conflicting or insufficient evidence. Third, we define a fine-grained evaluation protocol with four scenarios that disentangle parametric memorization from retrieval robustness. Experiments across 20 state-of-the-art LVLMs indicate that models usually fail to deflect in the presence of noisy or misleading evidence. Our results highlight the need to evaluate not only what models know, but how they behave when they do not, and serve as a reusable and extensible benchmark for reliable KB-VQA evaluation. All resources will be publicly available upon publication.