SokoBench: Evaluating Long-Horizon Planning and Reasoning in Large Language Models

TL;DR

This paper systematically evaluates the long-horizon planning and reasoning abilities of large language models using a Sokoban-based benchmark, revealing fundamental limitations in their planning capacity beyond 25 moves.

Contribution

It introduces a novel Sokoban-based benchmark for assessing long-horizon planning in large models and analyzes their performance limitations and potential improvements.

Findings

Performance degrades significantly beyond 25 moves

PDDL integration offers modest improvements

Architectural constraints limit planning capabilities

Abstract

Although the capabilities of large language models have been increasingly tested on complex reasoning tasks, their long-horizon planning abilities have not yet been extensively investigated. In this work, we provide a systematic assessment of the planning and long-horizon reasoning capabilities of state-of-the-art Large Reasoning Models (LRMs). We propose a novel benchmark based on Sokoban puzzles, intentionally simplified to isolate long-horizon planning from state persistence. Our findings reveal a consistent degradation in planning performance when more than 25 moves are required to reach the solution, suggesting a fundamental constraint on forward planning capacity. We show that equipping LRMs with Planning Domain Definition Language (PDDL) parsing, validation, and solving tools allows for modest improvements, suggesting inherent architectural limitations which might not be overcome by test-time scaling approaches alone.