Limited Reasoning Space: The cage of long-horizon reasoning in LLMs

TL;DR

This paper investigates the limitations of large language models in long-horizon reasoning, revealing that excessive planning can impair performance, and introduces Halo, a control framework that optimizes reasoning boundaries for better task-solving.

Contribution

The paper introduces Halo, a novel model predictive control framework that dynamically manages reasoning scope in LLMs, addressing over-planning issues in long-horizon tasks.

Findings

Halo outperforms static planning baselines on complex tasks.

Over-planning can lead to performance collapse in LLM reasoning.

Dynamic regulation of reasoning boundaries improves long-horizon reasoning.

Abstract

The test-time compute strategy, such as Chain-of-Thought (CoT), has significantly enhanced the ability of large language models to solve complex tasks like logical reasoning. However, empirical studies indicate that simply increasing the compute budget can sometimes lead to a collapse in test-time performance when employing typical task decomposition strategies such as CoT. This work hypothesizes that reasoning failures with larger compute budgets stem from static planning methods, which hardly perceive the intrinsic boundaries of LLM reasoning. We term it as the Limited Reasoning Space hypothesis and perform theoretical analysis through the lens of a non-autonomous stochastic dynamical system. This insight suggests that there is an optimal range for compute budgets; over-planning can lead to redundant feedback and may even impair reasoning capabilities. To exploit the compute-scaling benefits and suppress over-planning, this work proposes Halo, a model predictive control framework for LLM planning. Halo is designed for long-horizon tasks with reason-based planning and crafts an entropy-driven dual controller, which adopts a Measure-then-Plan strategy to achieve controllable reasoning. Experimental results demonstrate that Halo outperforms static baselines on complex long-horizon tasks by dynamically regulating planning at the reasoning boundary.