SIMS-V: Simulated Instruction-Tuning for Spatial Video Understanding

TL;DR

SIMS-V introduces a simulation-based data generation framework for training multimodal language models in spatial video understanding, enabling efficient transfer to real-world tasks with fewer data and question types.

Contribution

The paper presents SIMS-V, a novel systematic data-generation framework using 3D simulators to improve spatial reasoning in multimodal models, reducing data requirements and enhancing transferability.

Findings

A minimal set of three question categories suffices for effective transfer.

A 7B-parameter model fine-tuned on 25K simulated examples outperforms larger baselines.

The approach maintains general video understanding while improving spatial reasoning on real-world benchmarks.

Abstract

Despite impressive high-level video comprehension, multimodal language models struggle with spatial reasoning across time and space. While current spatial training approaches rely on real-world video data, obtaining diverse footage with precise spatial annotations remains a bottleneck. To alleviate this bottleneck, we present SIMS-V -- a systematic data-generation framework that leverages the privileged information of 3D simulators to create spatially-rich video training data for multimodal language models. Using this framework, we investigate which properties of simulated data drive effective real-world transfer through systematic ablations of question types, mixes, and scales. We identify a minimal set of three question categories (metric measurement, perspective-dependent reasoning, and temporal tracking) that prove most effective for developing transferable spatial intelligence, outperforming comprehensive coverage despite using fewer question types. These insights enable highly efficient training: our 7B-parameter video LLM fine-tuned on just 25K simulated examples outperforms the larger 72B baseline and achieves competitive performance with proprietary models on rigorous real-world spatial reasoning benchmarks. Our approach demonstrates robust generalization, maintaining performance on general video understanding while showing substantial improvements on embodied and real-world spatial tasks.