FaithSCAN: Model-Driven Single-Pass Hallucination Detection for Faithful Visual Question Answering

TL;DR

FaithSCAN is a lightweight, model-driven approach that detects hallucinations in visual question answering by leveraging internal signals of vision-language models, improving accuracy and efficiency without external resources.

Contribution

The paper introduces FaithSCAN, a novel internal-signal-based hallucination detection method for VQA that outperforms existing external and uncertainty-driven approaches.

Findings

FaithSCAN significantly improves detection accuracy over existing methods.

Internal signals reveal systematic causes of hallucinations in VQA models.

Hallucination patterns differ across VLM architectures, providing new insights.

Abstract

Faithfulness hallucinations in VQA occur when vision-language models produce fluent yet visually ungrounded answers, severely undermining their reliability in safety-critical applications. Existing detection methods mainly fall into two categories: external verification approaches relying on auxiliary models or knowledge bases, and uncertainty-driven approaches using repeated sampling or uncertainty estimates. The former suffer from high computational overhead and are limited by external resource quality, while the latter capture only limited facets of model uncertainty and fail to sufficiently explore the rich internal signals associated with the diverse failure modes. Both paradigms thus have inherent limitations in efficiency, robustness, and detection performance. To address these challenges, we propose FaithSCAN: a lightweight network that detects hallucinations by exploiting rich internal signals of VLMs, including token-level decoding uncertainty, intermediate visual representations, and cross-modal alignment features. These signals are fused via branch-wise evidence encoding and uncertainty-aware attention. We also extend the LLM-as-a-Judge paradigm to VQA hallucination and propose a low-cost strategy to automatically generate model-dependent supervision signals, enabling supervised training without costly human labels while maintaining high detection accuracy. Experiments on multiple VQA benchmarks show that FaithSCAN significantly outperforms existing methods in both effectiveness and efficiency. In-depth analysis shows hallucinations arise from systematic internal state variations in visual perception, cross-modal reasoning, and language decoding. Different internal signals provide complementary diagnostic cues, and hallucination patterns vary across VLM architectures, offering new insights into the underlying causes of multimodal hallucinations.