Exploring and Improving the Spatial Reasoning Abilities of Large Language Models

TL;DR

This paper evaluates the spatial reasoning abilities of large language models on 3D robotic trajectory data and introduces a prefix-based prompting method that significantly improves their performance.

Contribution

It provides the first detailed analysis of LLMs' spatial reasoning on 3D data and proposes a novel prompting technique that enhances their reasoning capabilities.

Findings

33% improvement on 3D trajectory data with new prompting

Up to 10% increase on SpartQA tasks

Insights into LLMs' engagement with spatial information

Abstract

Large Language Models (LLMs) represent formidable tools for sequence modeling, boasting an innate capacity for general pattern recognition. Nevertheless, their broader spatial reasoning capabilities, especially applied to numerical trajectory data, remain insufficiently explored. In this paper, we investigate the out-of-the-box performance of ChatGPT-3.5, ChatGPT-4 and Llama 2 7B models when confronted with 3D robotic trajectory data from the CALVIN baseline and associated tasks, including 2D directional and shape labeling. Additionally, we introduce a novel prefix-based prompting mechanism, which yields a 33% improvement on the 3D trajectory data and an increase of up to 10% on SpartQA tasks over zero-shot prompting (with gains for other prompting types as well). The experimentation with 3D trajectory data offers an intriguing glimpse into the manner in which LLMs engage with numerical and spatial information, thus laying a solid foundation for the identification of target areas for future enhancements.