ENTER: Event Based Interpretable Reasoning for VideoQA

TL;DR

ENTER introduces an interpretable VideoQA system using event graphs that enhance reasoning, robustness, and explainability in video question answering tasks.

Contribution

The paper proposes a novel event graph-based framework for VideoQA that improves interpretability, incorporates contextual visual information, and enhances robustness over existing methods.

Findings

Outperforms existing top-down approaches on multiple datasets.

Achieves competitive performance with bottom-up methods.

Provides superior interpretability and explainability in reasoning.

Abstract

In this paper, we present ENTER, an interpretable Video Question Answering (VideoQA) system based on event graphs. Event graphs convert videos into graphical representations, where video events form the nodes and event-event relationships (temporal/causal/hierarchical) form the edges. This structured representation offers many benefits: 1) Interpretable VideoQA via generated code that parses event-graph; 2) Incorporation of contextual visual information in the reasoning process (code generation) via event graphs; 3) Robust VideoQA via Hierarchical Iterative Update of the event graphs. Existing interpretable VideoQA systems are often top-down, disregarding low-level visual information in the reasoning plan generation, and are brittle. While bottom-up approaches produce responses from visual data, they lack interpretability. Experimental results on NExT-QA, IntentQA, and EgoSchema demonstrate that not only does our method outperform existing top-down approaches while obtaining competitive performance against bottom-up approaches, but more importantly, offers superior interpretability and explainability in the reasoning process.