MMSI-Video-Bench: A Holistic Benchmark for Video-Based Spatial Intelligence

TL;DR

MMSI-Video-Bench is a comprehensive, human-annotated benchmark designed to evaluate and advance spatial understanding capabilities of multimodal large language models in video contexts, revealing significant gaps and challenges.

Contribution

This work introduces MMSI-Video-Bench, a holistic, multi-level benchmark for video-based spatial intelligence, including diverse datasets, expert annotations, and targeted sub-benchmarks for comprehensive model evaluation.

Findings

Many models perform near chance levels.

The best reasoning models lag humans by nearly 60%.

Spatially fine-tuned models do not generalize well.

Abstract

Spatial understanding over continuous visual input is crucial for MLLMs to evolve into general-purpose assistants in physical environments. Yet there is still no comprehensive benchmark that holistically assesses the progress toward this goal. In this work, we introduce MMSI-Video-Bench, a fully human-annotated benchmark for video-based spatial intelligence in MLLMs. It operationalizes a four-level framework, Perception, Planning, Prediction, and Cross-Video Reasoning, through 1,106 questions grounded in 1,278 clips from 25 datasets and in-house videos. Each item is carefully designed and reviewed by 3DV experts with explanatory rationales to ensure precise, unambiguous grounding. Leveraging its diverse data sources and holistic task coverage, MMSI-Video-Bench also supports three domain-oriented sub-benchmarks (Indoor Scene Perception Bench, Robot Bench and Grounding Bench) for targeted capability assessment. We evaluate 25 strong open-source and proprietary MLLMs, revealing a striking human--AI gap: many models perform near chance, and the best reasoning model lags humans by nearly 60%. We further find that spatially fine-tuned models still fail to generalize effectively on our benchmark. Fine-grained error analysis exposes systematic failures in geometric reasoning, motion grounding, long-horizon prediction, and cross-video correspondence. We also show that typical frame-sampling strategies transfer poorly to our reasoning-intensive benchmark, and that neither 3D spatial cues nor chain-of-thought prompting yields meaningful gains. We expect our benchmark to establish a solid testbed for advancing video-based spatial intelligence.