Jigsaw-Puzzles: From Seeing to Understanding to Reasoning in Vision-Language Models

TL;DR

Jigsaw-Puzzles is a new benchmark with 1,100 complex images designed to evaluate and challenge the spatial reasoning and structural understanding capabilities of vision-language models, revealing significant performance gaps compared to humans.

Contribution

The paper introduces Jigsaw-Puzzles, a novel benchmark dataset and evaluation framework specifically targeting spatial reasoning in vision-language models, highlighting current limitations.

Findings

State-of-the-art VLMs achieve only 77.14% accuracy on the benchmark.

Models perform poorly on the Order Generation task, with only 30% accuracy.

There is a significant gap between model performance and human performance, emphasizing the need for further research.

Abstract

Spatial reasoning is a core component of human cognition, enabling individuals to perceive, comprehend, and interact with the physical world. It relies on a nuanced understanding of spatial structures and inter-object relationships, serving as the foundation for complex reasoning and decision-making. To investigate whether current vision-language models (VLMs) exhibit similar capability, we introduce Jigsaw-Puzzles, a novel benchmark consisting of 1,100 carefully curated real-world images with high spatial complexity. Based on this dataset, we design five tasks to rigorously evaluate VLMs' spatial perception, structural understanding, and reasoning capabilities, while deliberately minimizing reliance on domain-specific knowledge to better isolate and assess the general spatial reasoning capability. We conduct a comprehensive evaluation across 24 state-of-the-art VLMs. The results show that even the strongest model, Gemini-2.5-Pro, achieves only 77.14% overall accuracy and performs particularly poorly on the Order Generation task, with only 30.00% accuracy, far below the performance exceeding 90% achieved by human participants. This persistent gap underscores the need for continued progress, positioning Jigsaw-Puzzles as a challenging and diagnostic benchmark for advancing spatial reasoning research in VLMs. Our project page is at https://zesen01.github.io/jigsaw-puzzles.