Video-MSR: Benchmarking Multi-hop Spatial Reasoning Capabilities of MLLMs

TL;DR

This paper introduces Video-MSR, a new benchmark for evaluating multi-hop spatial reasoning in dynamic videos, revealing current model limitations and improving capabilities through specialized instruction tuning.

Contribution

The paper presents the first benchmark for multi-hop spatial reasoning in videos and demonstrates how instruction tuning enhances model performance on this challenging task.

Findings

Models perform well on perception but struggle with multi-step spatial reasoning.

Significant performance drops observed in MSR tasks across state-of-the-art models.

Fine-tuning with MSR-9K improves model accuracy by 7.82%.

Abstract

Spatial reasoning has emerged as a critical capability for Multimodal Large Language Models (MLLMs), drawing increasing attention and rapid advancement. However, existing benchmarks primarily focus on single-step perception-to-judgment tasks, leaving scenarios requiring complex visual-spatial logical chains significantly underexplored. To bridge this gap, we introduce Video-MSR, the first benchmark specifically designed to evaluate Multi-hop Spatial Reasoning (MSR) in dynamic video scenarios. Video-MSR systematically probes MSR capabilities through four distinct tasks: Constrained Localization, Chain-based Reference Retrieval, Route Planning, and Counterfactual Physical Deduction. Our benchmark comprises 3,052 high-quality video instances with 4,993 question-answer pairs, constructed via a scalable, visually-grounded pipeline combining advanced model generation with rigorous human verification. Through a comprehensive evaluation of 20 state-of-the-art MLLMs, we uncover significant limitations, revealing that while models demonstrate proficiency in surface-level perception, they exhibit distinct performance drops in MSR tasks, frequently suffering from spatial disorientation and hallucination during multi-step deductions. To mitigate these shortcomings and empower models with stronger MSR capabilities, we further curate MSR-9K, a specialized instruction-tuning dataset, and fine-tune Qwen-VL, achieving a +7.82% absolute improvement on Video-MSR. Our results underscore the efficacy of multi-hop spatial instruction data and establish Video-MSR as a vital foundation for future research. The code and data will be available at https://github.com/ruiz-nju/Video-MSR.