SpatiaLQA: A Benchmark for Evaluating Spatial Logical Reasoning in Vision-Language Models

TL;DR

This paper introduces SpatiaLQA, a benchmark for evaluating spatial logical reasoning in vision-language models, revealing current models' limitations and proposing a recursive scene graph method to improve reasoning capabilities.

Contribution

The paper presents SpatiaLQA, a new benchmark for spatial logical reasoning, and proposes a recursive scene graph method to enhance VLMs' reasoning abilities.

Findings

Most advanced VLMs struggle with spatial logical reasoning.

Recursive scene graph reasoning improves VLM performance.

SpatiaLQA provides a comprehensive evaluation dataset.

Abstract

Vision-Language Models (VLMs) have been increasingly applied in real-world scenarios due to their outstanding understanding and reasoning capabilities. Although VLMs have already demonstrated impressive capabilities in common visual question answering and logical reasoning, they still lack the ability to make reasonable decisions in complex real-world environments. We define this ability as spatial logical reasoning, which not only requires understanding the spatial relationships among objects in complex scenes, but also the logical dependencies between steps in multi-step tasks. To bridge this gap, we introduce Spatial Logical Question Answering (SpatiaLQA), a benchmark designed to evaluate the spatial logical reasoning capabilities of VLMs. SpatiaLQA consists of 9,605 question answer pairs derived from 241 real-world indoor scenes. We conduct extensive experiments on 41 mainstream VLMs, and the results show that even the most advanced models still struggle with spatial logical reasoning. To address this issue, we propose a method called recursive scene graph assisted reasoning, which leverages visual foundation models to progressively decompose complex scenes into task-relevant scene graphs, thereby enhancing the spatial logical reasoning ability of VLMs, outperforming all previous methods. Code and dataset are available at https://github.com/xieyc99/SpatiaLQA.