Milestone-Guided Policy Learning for Long-Horizon Language Agents

TL;DR

This paper introduces BEACON, a milestone-guided policy learning framework that improves training of long-horizon language agents by addressing credit misattribution and sample inefficiency, leading to significant performance gains.

Contribution

BEACON leverages task milestones for precise credit assignment, enhancing learning efficiency and success rates in long-horizon language agent tasks.

Findings

BEACON achieves 92.9% success on ALFWorld long-horizon tasks.

It nearly doubles the success rate compared to previous methods.

Sample utilization improves from 23.7% to 82.0% with BEACON.

Abstract

While long-horizon agentic tasks require language agents to perform dozens of sequential decisions, training such agents with reinforcement learning remains challenging. We identify two root causes: credit misattribution, where correct early actions are penalized due to terminal failures, and sample inefficiency, where scarce successful trajectories result in near-total loss of learning signal. We introduce a milestone-guided policy learning framework, BEACON, that leverages the compositional structure of long-horizon tasks to ensure precise credit assignment. BEACON partitions trajectories at milestone boundaries, applies temporal reward shaping within segments to credit partial progress, and estimates advantages at dual scales to prevent distant failures from corrupting the evaluation of local actions. On ALFWorld, WebShop, and ScienceWorld, BEACON consistently outperforms GRPO and GiGPO. Notably, on long-horizon ALFWorld tasks, BEACON achieves 92.9% success rate, nearly doubling GRPO's 53.5%, while improving effective sample utilization from 23.7% to 82.0%. These results establish milestone-anchored credit assignment as an effective paradigm for training long-horizon language agents. Code is available at https://github.com/ZJU-REAL/BEACON.