GEASS: Gated Evidence-Adaptive Selective Caption Trust for Vision-Language Models

TL;DR

GEASS is a training-free module that adaptively gates and weights captions during inference to improve vision-language model accuracy by reducing hallucinations and leveraging evidence-based trust.

Contribution

It introduces GEASS, a novel evidence-adaptive gating method that dynamically controls caption influence during inference without additional training.

Findings

GEASS improves accuracy on HallusionBench and POPE datasets.

It outperforms vanilla inference and contrastive decoding methods.

GEASS requires only two extra forward passes per query.

Abstract

Vision-Language Models (VLMs) excel at grounded reasoning but remain prone to object hallucination. Recent work treats self-generated captions as a uniformly positive resource, yet we find that naively embedding one can degrade rather than help--dropping Qwen2.5-VL-3B accuracy on HallusionBench by nearly 10 points. Two structural properties explain this. First, captions anchor not only the model's final answer but also its reasoning trajectory and lexical choices. Second, caption errors are asymmetric: omissions vastly outnumber fabrications, yet each fabrication carries a much larger per-instance impact. A caption's usefulness is therefore a per-query property, not a per-corpus one. We propose GEASS (ated Evidence-Adaptive Selective Caption Trust ), a training-free module that decides on each query how much of the caption the model consumes: it gates the caption by the clean path's confidence, weights it by the entropy reduction it produces, and raises the evidence bar when the two pathways disagree. Experiments on POPE and HallusionBench across four VLMs show that GEASS consistently improves over vanilla inference and contrastive decoding, with only two extra forward passes per query.