FoREST: Frame of Reference Evaluation in Spatial Reasoning Tasks

TL;DR

This paper introduces FoREST, a benchmark for evaluating large language models' understanding of the Frame of Reference in spatial reasoning, revealing performance gaps and proposing a spatial-guided prompting method to enhance comprehension.

Contribution

The paper presents the first dedicated FoR benchmark and a novel prompting method to improve spatial reasoning in LLMs.

Findings

Significant performance gaps across FoR classes in LLMs.

Spatial-Guided prompting improves spatial reasoning accuracy.

FoREST reveals critical shortcomings in current LLM spatial understanding.

Abstract

Spatial reasoning is a fundamental aspect of human intelligence. One key concept in spatial cognition is the Frame of Reference, which identifies the perspective of spatial expressions. Despite its significance, FoR has received limited attention in AI models that need spatial intelligence. There is a lack of dedicated benchmarks and in-depth evaluation of large language models (LLMs) in this area. To address this issue, we introduce the Frame of Reference Evaluation in Spatial Reasoning Tasks (FoREST) benchmark, designed to assess FoR comprehension in LLMs. We evaluate LLMs on answering questions that require FoR comprehension and layout generation in text-to-image models using FoREST. Our results reveal a notable performance gap across different FoR classes in various LLMs, affecting their ability to generate accurate layouts for text-to-image generation. This highlights critical shortcomings in FoR comprehension. To improve FoR understanding, we propose Spatial-Guided prompting, which improves LLMs ability to extract essential spatial concepts. Our proposed method improves overall performance across spatial reasoning tasks.