From Indoor to Open World: Revealing the Spatial Reasoning Gap in MLLMs

TL;DR

This paper introduces a large-scale outdoor spatial reasoning benchmark for Multimodal Large Language Models, revealing their reliance on linguistic priors over grounded visual reasoning, and highlights the gap in their spatial intelligence.

Contribution

The paper presents a novel outdoor dataset with metric ground truth for spatial reasoning, enabling comprehensive evaluation of MLLMs' spatial understanding in open-world scenarios.

Findings

MLLMs perform poorly in open-world spatial reasoning tasks.

Current models rely heavily on linguistic priors rather than visual grounding.

Structured indoor benchmarks do not reflect outdoor spatial reasoning capabilities.

Abstract

While Multimodal Large Language Models (MLLMs) have achieved impressive performance on semantic tasks, their spatial intelligence--crucial for robust and grounded AI systems--remains underdeveloped. Existing benchmarks fall short of diagnosing this limitation: they either focus on overly simplified qualitative reasoning or rely on domain-specific indoor data, constrained by the lack of outdoor datasets with verifiable metric ground truth. To bridge this gap, we introduce a large-scale benchmark built from pedestrian-perspective videos captured with synchronized stereo cameras, LiDAR, and IMU/GPS sensors. This dataset provides metrically precise 3D information, enabling the automatic generation of spatial reasoning questions that span a hierarchical spectrum--from qualitative relational reasoning to quantitative metric and kinematic understanding. Evaluations reveal that the performance gains observed in structured indoor benchmarks vanish in open-world settings. Further analysis using synthetic abnormal scenes and blinding tests confirms that current MLLMs depend heavily on linguistic priors instead of grounded visual reasoning. Our benchmark thus provides a principled platform for diagnosing these limitations and advancing physically grounded spatial intelligence.