Inherent limitations of LLMs regarding spatial information

TL;DR

This paper examines the limitations of large language models like ChatGPT in understanding and reasoning about spatial information, highlighting their underperformance in 2D and 3D navigation tasks.

Contribution

The study introduces a novel evaluation framework and a baseline dataset specifically designed to assess spatial reasoning in large language models.

Findings

ChatGPT struggles with spatial plotting and route planning tasks.

The dataset reveals significant gaps in spatial understanding of LLMs.

Insights into the specific limitations of LLMs in navigation tasks.

Abstract

Despite the significant advancements in natural language processing capabilities demonstrated by large language models such as ChatGPT, their proficiency in comprehending and processing spatial information, especially within the domains of 2D and 3D route planning, remains notably underdeveloped. This paper investigates the inherent limitations of ChatGPT and similar models in spatial reasoning and navigation-related tasks, an area critical for applications ranging from autonomous vehicle guidance to assistive technologies for the visually impaired. In this paper, we introduce a novel evaluation framework complemented by a baseline dataset, meticulously crafted for this study. This dataset is structured around three key tasks: plotting spatial points, planning routes in two-dimensional (2D) spaces, and devising pathways in three-dimensional (3D) environments. We specifically developed this dataset to assess the spatial reasoning abilities of ChatGPT. Our evaluation reveals key insights into the model's capabilities and limitations in spatial understanding.