Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models

TL;DR

This paper introduces SpatialEval, a new benchmark for evaluating spatial reasoning in vision-language models, revealing challenges and insights into how these models process spatial information and how multimodal redundancy can improve performance.

Contribution

The paper presents SpatialEval, a comprehensive benchmark for spatial reasoning, and provides an in-depth evaluation of models, uncovering surprising limitations and the impact of multimodal redundancy.

Findings

Spatial reasoning remains challenging for current models.

VLMs often underperform compared to LLMs despite visual input.

Redundancy between vision and text improves model performance.

Abstract

Large language models (LLMs) and vision-language models (VLMs) have demonstrated remarkable performance across a wide range of tasks and domains. Despite this promise, spatial understanding and reasoning -- a fundamental component of human cognition -- remains under-explored. We propose SpatialEval, a novel benchmark that covers diverse aspects of spatial reasoning such as relationship understanding, navigation, and counting. We conduct a comprehensive evaluation of competitive language and vision-language models. Our findings reveal several counter-intuitive insights that have been overlooked in the literature: (1) Spatial reasoning poses significant challenges where competitive models can fall behind random guessing; (2) Despite additional visual input, VLMs often under-perform compared to their LLM counterparts; (3) When both textual and visual information is available, multi-modal language models become less reliant on visual information if sufficient textual clues are provided. Additionally, we demonstrate that leveraging redundancy between vision and text can significantly enhance model performance. We hope our study will inform the development of multimodal models to improve spatial intelligence and further close the gap with human intelligence.