Motion-Grounded Video Reasoning: Understanding and Perceiving Motion at Pixel Level

TL;DR

This paper introduces a new task called Motion-Grounded Video Reasoning that requires models to generate pixel-level visual answers based on questions, advancing the understanding of motion and spatiotemporal reasoning in videos.

Contribution

It proposes a novel task, creates the GROUNDMORE dataset with diverse question types, and develops the MORA baseline model to enhance motion reasoning at the pixel level.

Findings

MORA outperforms existing models by 21.5% on GROUNDMORE.

GROUNDMORE contains 249K object masks across 1,715 videos with diverse question types.

The task advances the capabilities of models in implicit motion reasoning and spatiotemporal grounding.

Abstract

In this paper, we introduce Motion-Grounded Video Reasoning, a new motion understanding task that requires generating visual answers (video segmentation masks) according to the input question, and hence needs implicit spatiotemporal reasoning and grounding. This task extends existing spatiotemporal grounding work focusing on explicit action/motion grounding, to a more general format by enabling implicit reasoning via questions. To facilitate the development of the new task, we collect a large-scale dataset called GROUNDMORE, which comprises 1,715 video clips, 249K object masks that are deliberately designed with 4 question types (Causal, Sequential, Counterfactual, and Descriptive) for benchmarking deep and comprehensive motion reasoning abilities. GROUNDMORE uniquely requires models to generate visual answers, providing a more concrete and visually interpretable response than plain texts. It evaluates models on both spatiotemporal grounding and reasoning, fostering to address complex challenges in motion-related video reasoning, temporal perception, and pixel-level understanding. Furthermore, we introduce a novel baseline model named Motion-Grounded Video Reasoning Assistant (MORA). MORA incorporates the multimodal reasoning ability from the Multimodal LLM, the pixel-level perception capability from the grounding model (SAM), and the temporal perception ability from a lightweight localization head. MORA achieves respectable performance on GROUNDMORE outperforming the best existing visual grounding baseline model by an average of 21.5% relatively. We hope this novel and challenging task will pave the way for future advancements in robust and general motion understanding via video reasoning segmentation