Comment on The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity

TL;DR

This paper critiques prior claims of reasoning model failures by identifying experimental design flaws, demonstrating that models can perform well when proper evaluation methods are used, emphasizing the importance of careful experimental setup.

Contribution

The paper reveals critical issues in previous reasoning model evaluations and proposes improved evaluation strategies that better reflect models' true reasoning capabilities.

Findings

Models perform well on Tower of Hanoi when output constraints are managed.

Evaluation frameworks can misclassify reasoning failures due to practical constraints.

Unsolvable problem instances in benchmarks can lead to incorrect failure assessments.

Abstract

Shojaee et al. (2025) report that Large Reasoning Models (LRMs) exhibit "accuracy collapse" on planning puzzles beyond certain complexity thresholds. We demonstrate that their findings primarily reflect experimental design limitations rather than fundamental reasoning failures. Our analysis reveals three critical issues: (1) Tower of Hanoi experiments risk exceeding model output token limits, with models explicitly acknowledging these constraints in their outputs; (2) The authors' automated evaluation framework fails to distinguish between reasoning failures and practical constraints, leading to misclassification of model capabilities; (3) Most concerningly, their River Crossing benchmarks include mathematically impossible instances for N > 5 due to insufficient boat capacity, yet models are scored as failures for not solving these unsolvable problems. When we control for these experimental artifacts, by requesting generating functions instead of exhaustive move lists, preliminary experiments across multiple models indicate high accuracy on Tower of Hanoi instances previously reported as complete failures. These findings highlight the importance of careful experimental design when evaluating AI reasoning capabilities.