Diagnosing Causal Reasoning in Vision-Language Models via Structured Relevance Graphs

TL;DR

This paper introduces structured relevance graphs to diagnose and improve causal reasoning in vision-language models, revealing that structural guidance enhances reasoning more than capacity alone.

Contribution

We propose Vision-Language Causal Graphs (VLCGs) and a diagnostic benchmark ViLCaR to evaluate and enhance causal reasoning in LVLMs.

Findings

Injecting structured relevance improves attribution accuracy.

LVLMs' reasoning limitations are due to structural guidance deficits.

Structured evaluation reveals reasoning gaps beyond answer correctness.

Abstract

Large Vision-Language Models (LVLMs) achieve strong performance on visual question answering benchmarks, yet often rely on spurious correlations rather than genuine causal reasoning. Existing evaluations primarily assess the correctness of the answers, making it unclear whether failures arise from limited reasoning capability or from misidentifying causally relevant information. We introduce Vision-Language Causal Graphs (VLCGs), a structured, query-conditioned representation that explicitly encodes causally relevant objects, attributes, relations, and scene-grounded assumptions. Building on this representation, we present ViLCaR, a diagnostic benchmark comprising tasks for Causal Attribution, Causal Inference, and Question Answering, along with graph-aligned evaluation metrics that assess relevance identification beyond final answer accuracy. Experiments in state-of-the-art LVLMs show that injecting structured relevance information significantly improves attribution and inference consistency compared to zero-shot and standard in-context learning. These findings suggest that current limitations in LVLM causal reasoning stem primarily from insufficient structural guidance rather than a lack of reasoning capacity.