InfoCausalQA:Can Models Perform Non-explicit Causal Reasoning Based on Infographic?

TL;DR

This paper introduces InfoCausalQA, a benchmark for evaluating causal reasoning in vision-language models using infographics, revealing current models' limited capabilities in genuine causal inference compared to humans.

Contribution

The paper presents a new benchmark, InfoCausalQA, with a dataset and tasks designed to assess causal reasoning in multimodal models, highlighting their current limitations.

Findings

Current VLMs perform poorly on causal reasoning tasks.

Models show larger gaps in semantic causal reasoning compared to quantitative.

Humans significantly outperform models in causal inference tasks.

Abstract

Recent advances in Vision-Language Models (VLMs) have demonstrated impressive capabilities in perception and reasoning. However, the ability to perform causal inference -- a core aspect of human cognition -- remains underexplored, particularly in multimodal settings. In this study, we introduce InfoCausalQA, a novel benchmark designed to evaluate causal reasoning grounded in infographics that combine structured visual data with textual context. The benchmark comprises two tasks: Task 1 focuses on quantitative causal reasoning based on inferred numerical trends, while Task 2 targets semantic causal reasoning involving five types of causal relations: cause, effect, intervention, counterfactual, and temporal. We manually collected 494 infographic-text pairs from four public sources and used GPT-4o to generate 1,482 high-quality multiple-choice QA pairs. These questions were then carefully revised by humans to ensure they cannot be answered based on surface-level cues alone but instead require genuine visual grounding. Our experimental results reveal that current VLMs exhibit limited capability in computational reasoning and even more pronounced limitations in semantic causal reasoning. Their significantly lower performance compared to humans indicates a substantial gap in leveraging infographic-based information for causal inference. Through InfoCausalQA, we highlight the need for advancing the causal reasoning abilities of multimodal AI systems.