Multimodal Spatial Reasoning in the Large Model Era: A Survey and Benchmarks

TL;DR

This survey reviews recent progress in multimodal spatial reasoning with large models, introduces benchmarks, and discusses tasks across 2D, 3D, and embodied AI, highlighting advances in multimodal modalities like audio and egocentric video.

Contribution

It provides a comprehensive categorization of multimodal spatial reasoning tasks, introduces open benchmarks, and discusses recent progress in large multimodal reasoning models.

Findings

Progress in multimodal large language models (MLLMs) for spatial reasoning

Introduction of open benchmarks for evaluation

Coverage of spatial tasks across 2D, 3D, and embodied AI

Abstract

Humans possess spatial reasoning abilities that enable them to understand spaces through multimodal observations, such as vision and sound. Large multimodal reasoning models extend these abilities by learning to perceive and reason, showing promising performance across diverse spatial tasks. However, systematic reviews and publicly available benchmarks for these models remain limited. In this survey, we provide a comprehensive review of multimodal spatial reasoning tasks with large models, categorizing recent progress in multimodal large language models (MLLMs) and introducing open benchmarks for evaluation. We begin by outlining general spatial reasoning, focusing on post-training techniques, explainability, and architecture. Beyond classical 2D tasks, we examine spatial relationship reasoning, scene and layout understanding, as well as visual question answering and grounding in 3D space. We also review advances in embodied AI, including vision-language navigation and action models. Additionally, we consider emerging modalities such as audio and egocentric video, which contribute to novel spatial understanding through new sensors. We believe this survey establishes a solid foundation and offers insights into the growing field of multimodal spatial reasoning. Updated information about this survey, codes and implementation of the open benchmarks can be found at https://github.com/zhengxuJosh/Awesome-Spatial-Reasoning.