SITE: towards Spatial Intelligence Thorough Evaluation

TL;DR

This paper introduces SITE, a comprehensive benchmark dataset for evaluating spatial intelligence in vision-language models across various visual modalities and SI factors, revealing current models' limitations and their relation to embodied AI skills.

Contribution

The paper presents SITE, a novel standardized benchmark dataset for assessing spatial intelligence in vision-language models, including new tasks on view-taking and dynamic scenes.

Findings

Leading models underperform compared to humans in spatial orientation.

A positive correlation exists between spatial reasoning ability and embodied AI performance.

The benchmark covers diverse visual modalities and SI factors, highlighting current model gaps.

Abstract

Spatial intelligence (SI) represents a cognitive ability encompassing the visualization, manipulation, and reasoning about spatial relationships, underpinning disciplines from neuroscience to robotics. We introduce SITE, a benchmark dataset towards SI Thorough Evaluation in a standardized format of multi-choice visual question-answering, designed to assess large vision-language models' spatial intelligence across diverse visual modalities (single-image, multi-image, and video) and SI factors (figural to environmental scales, spatial visualization and orientation, intrinsic and extrinsic, static and dynamic). Our approach to curating the benchmark combines a bottom-up survey about 31 existing datasets and a top-down strategy drawing upon three classification systems in cognitive science, which prompt us to design two novel types of tasks about view-taking and dynamic scenes. Extensive experiments reveal that leading models fall behind human experts especially in spatial orientation, a fundamental SI factor. Moreover, we demonstrate a positive correlation between a model's spatial reasoning proficiency and its performance on an embodied AI task.