SPaRC: A Spatial Pathfinding Reasoning Challenge

TL;DR

SPaRC introduces a challenging dataset of 2D grid puzzles to evaluate and improve models' spatial and symbolic reasoning capabilities, highlighting current models' limitations in complex pathfinding tasks.

Contribution

The paper presents SPaRC, a new dataset for assessing multi-step spatial reasoning, and analyzes the shortcomings of existing models in solving complex pathfinding puzzles.

Findings

Humans achieve near-perfect accuracy on SPaRC puzzles.

Current models perform poorly, especially on hard puzzles.

Models often generate invalid paths and fail to scale with difficulty.

Abstract

Existing reasoning datasets saturate and fail to test abstract, multi-step problems, especially pathfinding and complex rule constraint satisfaction. We introduce SPaRC (Spatial Pathfinding Reasoning Challenge), a dataset of 1,000 2D grid pathfinding puzzles to evaluate spatial and symbolic reasoning, requiring step-by-step planning with arithmetic and geometric rules. Humans achieve near-perfect accuracy (98.0%; 94.5% on hard puzzles), while the best reasoning models, such as o4-mini, struggle (15.8%; 1.1% on hard puzzles). Models often generate invalid paths (>50% of puzzles for o4-mini), and reasoning tokens reveal they make errors in navigation and spatial logic. Unlike humans, who take longer on hard puzzles, models fail to scale test-time compute with difficulty. Allowing models to make multiple solution attempts improves accuracy, suggesting potential for better spatial reasoning with improved training and efficient test-time scaling methods. SPaRC can be used as a window into models' spatial reasoning limitations and drive research toward new methods that excel in abstract, multi-step problem-solving.