SAT: Dynamic Spatial Aptitude Training for Multimodal Language Models

TL;DR

This paper introduces SAT, a large simulated dataset for training multimodal language models to improve their understanding of both static and dynamic spatial relationships, with promising results on real-world tasks.

Contribution

The paper presents SAT, a novel large-scale simulated dataset for dynamic spatial reasoning, and demonstrates its effectiveness in enhancing multimodal models' spatial awareness, especially in dynamic scenarios.

Findings

Simulation-based training improves spatial reasoning in models.

Perfect annotations in simulation outperform pseudo-annotations.

Models trained with SAT outperform some proprietary models on real-world spatial tasks.

Abstract

Reasoning about motion and space is a fundamental cognitive capability that is required by multiple real-world applications. While many studies highlight that large multimodal language models (MLMs) struggle to reason about space, they only focus on static spatial relationships, and not dynamic awareness of motion and space, i.e., reasoning about the effect of egocentric and object motions on spatial relationships. Manually annotating such object and camera movements is expensive. Hence, we introduce SAT, a simulated spatial aptitude training dataset utilizing 3D simulators, comprising both static and dynamic spatial reasoning across 175K question-answer (QA) pairs and 20K scenes. Complementing this, we also construct a small (150 image-QAs) yet challenging dynamic spatial test set using real-world images. Leveraging our SAT datasets and 6 existing static spatial benchmarks, we systematically investigate what improves both static and dynamic spatial awareness. Our results reveal that simulations are surprisingly effective at imparting spatial aptitude to MLMs that translate to real images. We show that perfect annotations in simulation are more effective than existing approaches of pseudo-annotating real images. For instance, SAT training improves a LLaVA-13B model by an average 11% and a LLaVA-Video-7B model by an average 8% on multiple spatial benchmarks, including our real-image dynamic test set and spatial reasoning on long videos -- even outperforming some large proprietary models. While reasoning over static relationships improves with synthetic training data, there is still considerable room for improvement for dynamic reasoning questions.