SpaCE-10: A Comprehensive Benchmark for Multimodal Large Language Models in Compositional Spatial Intelligence

TL;DR

SpaCE-10 is a new benchmark designed to evaluate the spatial intelligence of multimodal large language models across atomic and compositional capabilities, revealing significant gaps compared to human performance.

Contribution

We introduce SpaCE-10, a comprehensive benchmark with a hierarchical annotation pipeline and extensive QA pairs to evaluate spatial reasoning in MLLMs.

Findings

Current MLLMs lag behind humans in spatial tasks.

Counting ability is a major limiting factor for MLLMs' spatial reasoning.

SpaCE-10 provides insights to improve MLLM spatial capabilities.

Abstract

Multimodal Large Language Models (MLLMs) have achieved remarkable progress in various multimodal tasks. To pursue higher intelligence in space, MLLMs require integrating multiple spatial capabilities, even for handling simple and normal tasks. However, existing benchmarks struggle to comprehensively evaluate the spatial intelligence of common MLLMs from the atomic level to the compositional level. To fill this gap, we present SpaCE-10, a comprehensive benchmark for compositional spatial evaluations. In SpaCE-10, we define 10 atomic spatial capabilities, which are combined to form 8 compositional capabilities. Based on these definitions, we propose a novel hierarchical annotation pipeline to generate high-quality and diverse question-answer (QA) pairs. With over 150+ hours of human expert effort, we obtain over 5k QA pairs for 811 real indoor scenes in SpaCE-10, which covers various evaluation settings like point cloud input and multi-choice QA. We conduct an extensive evaluation of common MLLMs on SpaCE-10 and find that even the most advanced MLLM still lags behind humans by large margins. Through our careful study, we also draw several significant findings that benefit the MLLM community. For example, we reveal that the shortcoming of counting capability greatly limits the compositional spatial capabilities of existing MLLMs.