Tool-Augmented Spatiotemporal Reasoning for Streamlining Video Question Answering Task
Sunqi Fan, Jiashuo Cui, Meng-Hao Guo, Shuojin Yang

TL;DR
This paper introduces a spatiotemporal reasoning framework and toolkit to improve multimodal large language models' ability to understand and reason about complex videos for question answering, achieving significant performance gains.
Contribution
It proposes a comprehensive Video Toolkit and a Spatiotemporal Reasoning Framework (STAR) to enhance MLLMs' reasoning capabilities in video question answering tasks.
Findings
Achieved 8.2% improvement on VideoMME
Achieved 4.6% improvement on LongVideoBench
Enhanced GPT-4o with lightweight tools for better reasoning
Abstract
Video Question Answering (VideoQA) task serves as a critical playground for evaluating whether foundation models can effectively perceive, understand, and reason about dynamic real-world scenarios. However, existing Multimodal Large Language Models (MLLMs) struggle with simultaneously modeling spatial relationships within video frames and understanding the causal dynamics of temporal evolution on complex and reasoning-intensive VideoQA task. In this work, we equip MLLM with a comprehensive and extensible Video Toolkit, to enhance MLLM's spatiotemporal reasoning capabilities and ensure the harmony between the quantity and diversity of tools. To better control the tool invocation sequence and avoid toolchain shortcut issues, we propose a Spatiotemporal Reasoning Framework (STAR) that strategically schedules temporal and spatial tools, thereby progressively localizing the key area in the…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMultimodal Machine Learning Applications · Topic Modeling · Explainable Artificial Intelligence (XAI)
