An Evaluation of ChatGPT-4's Qualitative Spatial Reasoning Capabilities in RCC-8

TL;DR

This paper evaluates ChatGPT-4's ability to perform classical qualitative spatial reasoning tasks using the RCC-8 calculus, highlighting its strengths and limitations in this domain.

Contribution

It provides the first systematic assessment of ChatGPT-4's capabilities in RCC-8 qualitative spatial reasoning tasks.

Findings

ChatGPT-4 demonstrates partial understanding of RCC-8 relations.

Performance varies significantly across different reasoning tasks.

The study highlights areas for improvement in LLM-based spatial reasoning.

Abstract

Qualitative Spatial Reasoning (QSR) is well explored area of Commonsense Reasoning and has multiple applications ranging from Geographical Information Systems to Robotics and Computer Vision. Recently many claims have been made for the capabilities of Large Language Models (LLMs). In this paper we investigate the extent to which one particular LLM can perform classical qualitative spatial reasoning tasks on the mereotopological calculus, RCC-8.