Evaluating the Ability of Large Language Models to Reason about Cardinal Directions, Revisited

TL;DR

This paper evaluates 28 large language models' ability to reason about cardinal directions using a comprehensive benchmark, revealing that even advanced models struggle with reliable reasoning in this domain.

Contribution

It introduces a new benchmark for testing LLMs on reasoning about cardinal directions, extending previous work and highlighting current limitations.

Findings

Most LLMs fail to reliably determine correct cardinal directions.

Even recent large reasoning models show significant reasoning errors.

The benchmark reveals persistent challenges in spatial reasoning for LLMs.

Abstract

We investigate the abilities of 28 Large language Models (LLMs) to reason about cardinal directions (CDs) using a benchmark generated from a set of templates, extensively testing an LLM's ability to determine the correct CD given a particular scenario. The templates allow for a number of degrees of variation such as means of locomotion of the agent involved, and whether set in the first, second or third person. Even the newer Large Reasoning Models are unable to reliably determine the correct CD for all questions. This paper summarises and extends earlier work presented at COSIT-24.