MazeEval: A Benchmark for Testing Sequential Decision-Making in Language Models

TL;DR

MazeEval introduces a benchmark to evaluate spatial reasoning in language models through coordinate-based maze navigation tasks, revealing disparities in performance across models and languages, and highlighting limitations in current LLMs' spatial cognition.

Contribution

This paper presents MazeEval, a novel benchmark for assessing pure spatial reasoning in LLMs without visual input, and provides insights into models' cross-linguistic and complexity-related performance limitations.

Findings

OpenAI's O3 achieves perfect navigation up to 30x30 mazes.

Most models fail beyond 9x9 mazes, often looping excessively.

Spatial reasoning performance degrades in Icelandic, indicating language-dependent learning.

Abstract

As Large Language Models (LLMs) increasingly power autonomous agents in robotics and embodied AI, understanding their spatial reasoning capabilities becomes crucial for ensuring reliable real-world deployment. Despite advances in language understanding, current research lacks evaluation of how LLMs perform spatial navigation without visual cues, a fundamental requirement for agents operating with limited sensory information. This paper addresses this gap by introducing MazeEval, a benchmark designed to isolate and evaluate pure spatial reasoning in LLMs through coordinate-based maze navigation tasks. Our methodology employs a function-calling interface where models navigate mazes of varying complexity ($5\times 5$ to $15\times 15$ grids) using only coordinate feedback and distance-to-wall information, excluding visual input to test fundamental spatial cognition. We evaluate eight state-of-the-art LLMs across identical mazes in both English and Icelandic to assess cross-linguistic transfer of spatial abilities. Our findings reveal striking disparities: while OpenAI's O3 achieves perfect navigation for mazes up to size $30\times 30$, other models exhibit catastrophic failure beyond $9\times 9$ mazes, with 100% of failures attributed to excessive looping behavior where models revisit a cell at least 10 times. We document a significant performance degradation in Icelandic, with models solving mazes 3-4 sizes smaller than in English, suggesting spatial reasoning in LLMs emerges from linguistic patterns rather than language-agnostic mechanisms. These results have important implications for global deployment of LLM-powered autonomous systems, showing spatial intelligence remains fundamentally constrained by training data availability and highlighting the need for architectural innovations to achieve reliable navigation across linguistic contexts.