MECD+: Unlocking Event-Level Causal Graph Discovery for Video Reasoning

TL;DR

This paper introduces MECD+, a novel framework and dataset for uncovering detailed event-level causal graphs in long videos, enhancing understanding of complex causal relationships beyond brief segments.

Contribution

It proposes a new task, dataset, and a Granger causality-inspired framework with advanced inference techniques for comprehensive causal discovery in videos.

Findings

Outperforms GPT-4o and VideoChat2 in causal reasoning accuracy.

Enables improved downstream video understanding tasks.

Effectively models complex, interconnected causal relations in videos.

Abstract

Video causal reasoning aims to achieve a high-level understanding of videos from a causal perspective. However, it exhibits limitations in its scope, primarily executed in a question-answering paradigm and focusing on brief video segments containing isolated events and basic causal relations, lacking comprehensive and structured causality analysis for videos with multiple interconnected events. To fill this gap, we introduce a new task and dataset, Multi-Event Causal Discovery (MECD). It aims to uncover the causal relations between events distributed chronologically across long videos. Given visual segments and textual descriptions of events, MECD identifies the causal associations between these events to derive a comprehensive and structured event-level video causal graph explaining why and how the result event occurred. To address the challenges of MECD, we devise a novel framework inspired by the Granger Causality method, incorporating an efficient mask-based event prediction model to perform an Event Granger Test. It estimates causality by comparing the predicted result event when premise events are masked versus unmasked. Furthermore, we integrate causal inference techniques such as front-door adjustment and counterfactual inference to mitigate challenges in MECD like causality confounding and illusory causality. Additionally, context chain reasoning is introduced to conduct more robust and generalized reasoning. Experiments validate the effectiveness of our framework in reasoning complete causal relations, outperforming GPT-4o and VideoChat2 by 5.77% and 2.70%, respectively. Further experiments demonstrate that causal relation graphs can also contribute to downstream video understanding tasks such as video question answering and video event prediction.