CounterVQA: Evaluating and Improving Counterfactual Reasoning in Vision-Language Models for Video Understanding

TL;DR

This paper introduces CounterVQA, a benchmark for evaluating counterfactual reasoning in video-language models, revealing current limitations and proposing CFGPT to improve reasoning capabilities through language-based distillation.

Contribution

The paper presents CounterVQA, a new benchmark for counterfactual reasoning in videos, and proposes CFGPT, a method to enhance models' reasoning by leveraging language modality.

Findings

Models perform poorly on complex counterfactual questions.

CounterVQA reveals significant gaps in current models' reasoning abilities.

CFGPT improves reasoning performance across all difficulty levels.

Abstract

Vision Language Models (VLMs) have recently shown significant advancements in video understanding, especially in feature alignment, event reasoning, and instruction-following tasks. However, their capability for counterfactual reasoning, inferring alternative outcomes under hypothetical conditions, remains underexplored. This capability is essential for robust video understanding, as it requires identifying underlying causal structures and reasoning about unobserved possibilities, rather than merely recognizing observed patterns. To systematically evaluate this capability, we introduce CounterVQA, a video-based benchmark featuring three progressive difficulty levels that assess different aspects of counterfactual reasoning. Through comprehensive evaluation of both state-of-the-art open-source and closed-source models, we uncover a substantial performance gap: while these models achieve reasonable accuracy on simple counterfactual questions, performance degrades significantly on complex multi-hop causal chains. To address these limitations, we develop a post-training method, CFGPT, that enhances a model's visual counterfactual reasoning ability by distilling its counterfactual reasoning capability from the language modality, yielding consistent improvements across all CounterVQA difficulty levels. Dataset and code will be further released.