Multi-SpatialMLLM: Multi-Frame Spatial Understanding with Multi-Modal Large Language Models

TL;DR

This paper introduces Multi-SpatialMLLM, a multi-modal large language model enhanced with multi-frame spatial understanding, leveraging a new large-scale dataset and benchmark to improve robotics and real-world scene reasoning.

Contribution

The paper presents a novel framework integrating depth, correspondence, and dynamic perception into MLLMs, along with the MultiSPA dataset and a comprehensive spatial reasoning benchmark.

Findings

Significant performance improvements over baselines.

Demonstrated scalability and generalization in multi-frame reasoning.

Emergent capabilities in complex scenarios.

Abstract

Multi-modal large language models (MLLMs) have rapidly advanced in visual tasks, yet their spatial understanding remains limited to single images, leaving them ill-suited for robotics and other real-world applications that require multi-frame reasoning. In this paper, we propose a framework to equip MLLMs with robust multi-frame spatial understanding by integrating depth perception, visual correspondence, and dynamic perception. Central to our approach is the MultiSPA dataset, a novel, large-scale collection of more than 27 million samples spanning diverse 3D and 4D scenes. Alongside MultiSPA, we introduce a comprehensive benchmark that tests a wide spectrum of spatial tasks under uniform metrics. Our resulting model, Multi-SpatialMLLM, achieves significant gains over baselines and proprietary systems, demonstrating scalable, generalizable multi-frame reasoning. We further observe multi-task benefits and early indications of emergent capabilities in challenging scenarios, and showcase how our model can serve as a multi-frame reward annotator for robotics.