SVAgent: Storyline-Guided Long Video Understanding via Cross-Modal Multi-Agent Collaboration

TL;DR

SVAgent introduces a multi-agent framework that models storyline reasoning for VideoQA, improving interpretability and performance by mimicking human-like video understanding.

Contribution

It presents a novel cross-modal multi-agent system that constructs storylines and enhances reasoning robustness in video question answering.

Findings

SVAgent outperforms existing methods on VideoQA benchmarks.

The framework improves interpretability by emulating human storyline reasoning.

Experimental results show increased answer accuracy and consistency.

Abstract

Video question answering (VideoQA) is a challenging task that requires integrating spatial, temporal, and semantic information to capture the complex dynamics of video sequences. Although recent advances have introduced various approaches for video understanding, most existing methods still rely on locating relevant frames to answer questions rather than reasoning through the evolving storyline as humans do. Humans naturally interpret videos through coherent storylines, an ability that is crucial for making robust and contextually grounded predictions. To address this gap, we propose SVAgent, a storyline-guided cross-modal multi-agent framework for VideoQA. The storyline agent progressively constructs a narrative representation based on frames suggested by a refinement suggestion agent that analyzes historical failures. In addition, cross-modal decision agents independently predict answers from visual and textual modalities under the guidance of the evolving storyline. Their outputs are then evaluated by a meta-agent to align cross-modal predictions and enhance reasoning robustness and answer consistency. Experimental results demonstrate that SVAgent achieves superior performance and interpretability by emulating human-like storyline reasoning in video understanding.