DO-Bench: An Attributable Benchmark for Diagnosing Object Hallucination in Vision-Language Models

TL;DR

DO-Bench is a diagnostic benchmark designed to disentangle perceptual and contextual causes of object hallucination in vision-language models, enabling targeted evaluation of their reliability.

Contribution

It introduces a structured, controlled framework with diagnostic metrics to attribute hallucination errors to perceptual or prior-related failures.

Findings

Models show systematic differences in prior sensitivity and perceptual reliability.

Object hallucination mechanisms vary across models, beyond what aggregate accuracy indicates.

Abstract

Object level hallucination remains a central reliability challenge for vision language models (VLMs), particularly in binary object existence verification. Existing benchmarks emphasize aggregate accuracy but rarely disentangle whether errors stem from perceptual limitations or from the influence of contextual textual priors, leaving underlying failure mechanisms ambiguous. We introduce DO-Bench, a controlled diagnostic benchmark that isolates these sources through structured multimodal interventions. Rather than evaluating models in unconstrained settings, DO-Bench probes two complementary dimensions: the Prior Override dimension progressively strengthens contextual textual priors while holding visual evidence constant to assess resistance to prior pressure, and the Perception-Limited dimension incrementally enhances visual evidence from full-scene context to localized object crops to measure perceptual grounding strength. This paired design enables attribution of errors to prior suppression, perceptual insufficiency, or their interaction. We further define two diagnostic metrics, PriorRobust and PerceptionAbility, to quantify these behaviors consistently. Evaluations across diverse open- and closed-source VLMs reveal systematic differences in prior sensitivity and perceptual reliability, demonstrating that object hallucination reflects heterogeneous, mechanism dependent failure patterns beyond aggregate accuracy.