DeepPlanning: Benchmarking Long-Horizon Agentic Planning with Verifiable Constraints

TL;DR

DeepPlanning introduces a challenging benchmark for long-horizon agent planning involving complex constraints, revealing current LLM limitations and guiding future improvements in explicit reasoning and tool use.

Contribution

The paper presents DeepPlanning, a new benchmark for practical long-horizon planning with real-world constraints, highlighting the need for improved reasoning and tool integration in agentic LLMs.

Findings

Current LLMs struggle with long-horizon planning tasks.

Explicit reasoning patterns improve planning effectiveness.

Parallel tool use enhances efficiency in complex tasks.

Abstract

While agent evaluation has shifted toward long-horizon tasks, most benchmarks still emphasize local, step-level reasoning rather than the global constrained optimization (e.g., time and financial budgets) that demands genuine planning ability. Meanwhile, existing LLM planning benchmarks underrepresent the active information gathering and fine-grained local constraints typical of real-world settings. To address this, we introduce DeepPlanning, a challenging benchmark for practical long-horizon agent planning. It features multi-day travel planning and multi-product shopping tasks that require proactive information acquisition, local constrained reasoning, and global constrained optimization. Evaluations on DeepPlanning show that even frontier agentic LLMs struggle with these problems, highlighting the importance of reliable explicit reasoning patterns and parallel tool use for achieving better effectiveness-efficiency trade-offs. Error analysis further points to promising directions for improving agentic LLMs over long planning horizons. We open-source the code and data to support future research.