PilotRL: Training Language Model Agents via Global Planning-Guided Progressive Reinforcement Learning

TL;DR

PilotRL introduces a global planning-guided reinforcement learning framework for LLM agents, enhancing long-term strategic decision-making and generalization in complex tasks beyond existing single-step reasoning methods.

Contribution

The paper proposes AdaPlan and PilotRL, novel paradigms that integrate explicit global planning with reinforcement learning to improve LLM agent performance and planning quality.

Findings

PilotRL achieves state-of-the-art performance surpassing GPT-4o by 3.60%.

Significant 55.78% improvement over GPT-4o-mini at similar scale.

Enhanced ability of LLMs to follow explicit global plans in agent tasks.

Abstract

Large Language Models (LLMs) have shown remarkable advancements in tackling agent-oriented tasks. Despite their potential, existing work faces challenges when deploying LLMs in agent-based environments. The widely adopted agent paradigm ReAct centers on integrating single-step reasoning with immediate action execution, which limits its effectiveness in complex tasks requiring long-term strategic planning. Furthermore, the coordination between the planner and executor during problem-solving is also a critical factor to consider in agent design. Additionally, current approaches predominantly rely on supervised fine-tuning, which often leads models to memorize established task completion trajectories, thereby restricting their generalization ability when confronted with novel problem contexts. To address these challenges, we introduce an adaptive global plan-based agent paradigm AdaPlan, aiming to synergize high-level explicit guidance with execution to support effective long-horizon decision-making. Based on the proposed paradigm, we further put forward PilotRL, a global planning-guided training framework for LLM agents driven by progressive reinforcement learning. We first develop the model's ability to follow explicit guidance from global plans when addressing agent tasks. Subsequently, based on this foundation, we focus on optimizing the quality of generated plans. Finally, we conduct joint optimization of the model's planning and execution coordination. Experiments indicate that PilotRL could achieve state-of-the-art performances, with LLaMA3.1-8B-Instruct + PilotRL surpassing closed-sourced GPT-4o by 3.60%, while showing a more substantial gain of 55.78% comparing to GPT-4o-mini at a comparable parameter scale.