LogicGaze: Benchmarking Causal Consistency in Visual Narratives via Counterfactual Verification

TL;DR

LogicGaze is a benchmark framework that tests vision-language models' ability to verify causal reasoning chains against visual evidence, revealing vulnerabilities and emphasizing the need for trustworthy multimodal reasoning.

Contribution

Introduces LogicGaze, a comprehensive benchmark with novel evaluation protocols to assess causal verification in vision-language models, addressing hallucination issues.

Findings

State-of-the-art VLMs show significant vulnerabilities in causal verification.

LogicGaze exposes hallucination and reasoning errors in models like Qwen2.5-VL-72B.

Benchmark resources are publicly available for further research.

Abstract

While sequential reasoning enhances the capability of Vision-Language Models (VLMs) to execute complex multimodal tasks, their reliability in grounding these reasoning chains within actual visual evidence remains insufficiently explored. We introduce LogicGaze, a novel benchmark framework designed to rigorously interrogate whether VLMs can validate sequential causal chains against visual inputs, specifically targeting the pervasive issue of hallucination. Curated from 40,000 video segments from ShareGPT4Video and a subset of Flickr30k imagery, LogicGaze integrates causal sequences with visually contradictory yet linguistically plausible perturbations, compelling models to verify the authenticity of each reasoning step. Our tripartite evaluation protocol - Causal Validation, Grounded Narrative Synthesis, and Perturbation Rejection - exposes significant vulnerabilities in state-of-the-art VLMs such as Qwen2.5-VL-72B. LogicGaze advocates for robust, trustworthy multimodal reasoning, with all resources publicly available in an anonymized repository.