Spatial4D-Bench: A Versatile 4D Spatial Intelligence Benchmark

TL;DR

Spatial4D-Bench is a comprehensive large-scale benchmark designed to evaluate the 4D spatial reasoning abilities of Multimodal Large Language Models, revealing significant limitations and guiding future development.

Contribution

This work introduces a versatile, multi-task 4D spatial intelligence benchmark with 40,000 questions across 18 tasks, filling a gap in existing spatial reasoning evaluation tools.

Findings

State-of-the-art MLLMs show substantial limitations in 4D spatial reasoning

Benchmark covers diverse tasks like route planning, action recognition, and physical plausibility

Provides insights to guide development of more capable MLLMs

Abstract

4D spatial intelligence involves perceiving and processing how objects move or change over time. Humans naturally possess 4D spatial intelligence, supporting a broad spectrum of spatial reasoning abilities. To what extent can Multimodal Large Language Models (MLLMs) achieve human-level 4D spatial intelligence? In this work, we present Spatial4D-Bench, a versatile 4D spatial intelligence benchmark designed to comprehensively assess the 4D spatial reasoning abilities of MLLMs. Unlike existing spatial intelligence benchmarks that are often small-scale or limited in diversity, Spatial4D-Bench provides a large-scale, multi-task evaluation benchmark consisting of ~40,000 question-answer pairs covering 18 well-defined tasks. We systematically organize these tasks into six cognitive categories: object understanding, scene understanding, spatial relationship understanding, spatiotemporal relationship understanding, spatial reasoning and spatiotemporal reasoning. Spatial4D-Bench thereby offers a structured and comprehensive benchmark for evaluating the spatial cognition abilities of MLLMs, covering a broad spectrum of tasks that parallel the versatility of human spatial intelligence. We benchmark various state-of-the-art open-source and proprietary MLLMs on Spatial4D-Bench and reveal their substantial limitations in a wide variety of 4D spatial reasoning aspects, such as route plan, action recognition, and physical plausibility reasoning. We hope that the findings provided in this work offer valuable insights to the community and that our benchmark can facilitate the development of more capable MLLMs toward human-level 4D spatial intelligence. More resources can be found on our project page.