Thinking in Structures: Evaluating Spatial Intelligence through Reasoning on Constrained Manifolds

TL;DR

This paper introduces SSI-Bench, a challenging VQA benchmark for spatial reasoning on constrained 3D structures, revealing significant gaps between current models and human performance.

Contribution

It presents SSI-Bench, a novel benchmark with complex real-world 3D structures for evaluating spatial reasoning in vision-language models, created through a human-centered process.

Findings

Current models perform poorly compared to humans on SSI-Bench.

Models show limited improvement even when encouraged to think.

Error analysis highlights failures in structural grounding and 3D reasoning.

Abstract

Spatial intelligence is crucial for vision--language models (VLMs) in the physical world, yet many benchmarks evaluate largely unconstrained scenes where models can exploit 2D shortcuts. We introduce SSI-Bench, a VQA benchmark for spatial reasoning on constrained manifolds, built from complex real-world 3D structures whose feasible configurations are tightly governed by geometric, topological, and physical constraints. SSI-Bench contains 1,000 ranking questions spanning geometric and topological reasoning and requiring a diverse repertoire of compositional spatial operations, such as mental rotation, cross-sectional inference, occlusion reasoning, and force-path reasoning. It is created via a fully human-centered pipeline: ten researchers spent over 400 hours curating images, annotating structural components, and designing questions to minimize pixel-level cues. Evaluating 31 widely used VLMs reveals a large gap to humans: the best open-source model achieves 22.2% accuracy and the strongest closed-source model reaches 33.6%, while humans score 91.6%. Encouraging models to think yields only marginal gains, and error analysis points to failures in structural grounding and constraint-consistent 3D reasoning. Project page: https://ssi-bench.github.io.