TopViewRS: Vision-Language Models as Top-View Spatial Reasoners

TL;DR

This paper evaluates the spatial reasoning abilities of vision-language models from a top-view perspective, revealing significant gaps compared to human performance and highlighting the need for improved reasoning capabilities.

Contribution

Introduces the TopViewRS dataset and systematically assesses VLMs' top-view spatial reasoning, exposing their limitations and guiding future research.

Findings

VLMs perform over 50% worse than humans on spatial reasoning tasks.

Chain-of-Thought reasoning improves VLM performance by 5.82%.

VLMs often perform worse than random baselines in some cases.

Abstract

Top-view perspective denotes a typical way in which humans read and reason over different types of maps, and it is vital for localization and navigation of humans as well as of `non-human' agents, such as the ones backed by large Vision-Language Models (VLMs). Nonetheless, spatial reasoning capabilities of modern VLMs remain unattested and underexplored. In this work, we thus study their capability to understand and reason over spatial relations from the top view. The focus on top view also enables controlled evaluations at different granularity of spatial reasoning; we clearly disentangle different abilities (e.g., recognizing particular objects versus understanding their relative positions). We introduce the TopViewRS (Top-View Reasoning in Space) dataset, consisting of 11,384 multiple-choice questions with either realistic or semantic top-view map as visual input. We then use it to study and evaluate VLMs across 4 perception and reasoning tasks with different levels of complexity. Evaluation of 10 representative open- and closed-source VLMs reveals the gap of more than 50% compared to average human performance, and it is even lower than the random baseline in some cases. Although additional experiments show that Chain-of-Thought reasoning can boost model capabilities by 5.82% on average, the overall performance of VLMs remains limited. Our findings underscore the critical need for enhanced model capability in top-view spatial reasoning and set a foundation for further research towards human-level proficiency of VLMs in real-world multimodal tasks.