Scaling Spatial Reasoning in MLLMs through Programmatic Data Synthesis

TL;DR

This paper introduces SPRITE, a framework that uses simulators and large models to generate high-quality, diverse spatial reasoning data, significantly improving the spatial understanding of vision-language models.

Contribution

SPRITE reframes ground-truth generation as a code-generation task, enabling scalable, diverse, and precise spatial reasoning data synthesis using simulators and large language models.

Findings

VLM trained on SPRITE data outperforms other datasets on spatial benchmarks.

SPRITE dataset includes over 300k instruction pairs from 3 simulators.

Scalability analysis confirms the importance of diverse data for robust spatial reasoning.

Abstract

Embodied intelligence, a grand challenge in artificial intelligence, is fundamentally constrained by the limited spatial understanding and reasoning capabilities of current models. Prevailing efforts to address this through enhancing Vision-Language Models (VLMs) are trapped in a dilemma: template-based datasets are scalable but structurally rigid, while manual annotation is linguistically diverse but unscalable and, critically, computationally imprecise. We introduce SPRITE, a novel framework that overcomes this dilemma by leveraging simulators and large models to programmatically synthesize scalable, diverse, and high-quality spatial reasoning data. The core innovation of SPRITE is to reframe ground-truth generation as a code-generation task. We utilize LLMs to compile complex spatial questions into executable programs, which are then verified against high-precision scene meta-information extracted from simulators. This ensures our ground truth is both computationally precise and verifiable, while the generative power of LLMs provides vast linguistic diversity. Leveraging this pipeline, we have curated a dataset encompassing 3 simulators, 11k+ scenes, and 300k+ image/video instruction-tuning pairs. We demonstrate that a VLM trained on our data achieves significant performance gains on multiple spatial benchmarks and outperforms other open-source datasets of equivalent size. Furthermore, a scalability analysis confirms our hypothesis that overcoming the low-diversity nature of traditional template methods is essential for building robust, generalizable spatial intelligence. We will make the SPRITE framework code and the full 300k+ dataset publicly available to facilitate future research in spatial intelligence.