FloorplanQA: A Benchmark for Spatial Reasoning in LLMs using Structured Representations

TL;DR

FloorplanQA is a benchmark designed to evaluate the spatial reasoning abilities of large-language models using structured indoor scene representations, revealing their limitations in understanding physical constraints and spatial coherence.

Contribution

We introduce FloorplanQA, a novel benchmark with structured representations to assess spatial reasoning in LLMs, highlighting their shortcomings in physical and spatial understanding.

Findings

Models succeed in shallow queries but struggle with physical constraints.

Models are robust to small spatial perturbations.

FloorplanQA exposes a blind spot in LLM spatial reasoning.

Abstract

We introduce FloorplanQA, a diagnostic benchmark for evaluating spatial reasoning in large-language models (LLMs). FloorplanQA is grounded in structured representations of indoor scenes, such as (e.g., kitchens, living rooms, bedrooms, bathrooms, and others), encoded symbolically in JSON or XML layouts. The benchmark covers core spatial tasks, including distance measurement, visibility, path finding, and object placement within constrained spaces. Our results across a variety of frontier open-source and commercial LLMs reveal that while models may succeed in shallow queries, they often fail to respect physical constraints, preserve spatial coherence, though they remain mostly robust to small spatial perturbations. FloorplanQA uncovers a blind spot in today's LLMs: inconsistent reasoning about indoor layouts. We hope this benchmark inspires new work on language models that can accurately infer and manipulate spatial and geometric properties in practical settings.