HiPER: Hierarchical Reinforcement Learning with Explicit Credit Assignment for Large Language Model Agents

TL;DR

HiPER introduces a hierarchical reinforcement learning framework for large language model agents, explicitly separating planning and execution, leading to improved performance on complex, long-horizon decision-making tasks.

Contribution

The paper proposes HiPER, a hierarchical RL approach with explicit credit assignment and a novel advantage estimation technique, enhancing training stability and efficiency for LLM agents.

Findings

Achieves state-of-the-art success rates on ALFWorld and WebShop benchmarks.

Outperforms prior methods by 6.6% and 8.3% respectively.

Significantly improves performance on long-horizon tasks.

Abstract

Training LLMs as interactive agents for multi-turn decision-making remains challenging, particularly in long-horizon tasks with sparse and delayed rewards, where agents must execute extended sequences of actions before receiving meaningful feedback. Most existing reinforcement learning (RL) approaches model LLM agents as flat policies operating at a single time scale, selecting one action at each turn. In sparse-reward settings, such flat policies must propagate credit across the entire trajectory without explicit temporal abstraction, which often leads to unstable optimization and inefficient credit assignment. We propose HiPER, a novel Hierarchical Plan-Execute RL framework that explicitly separates high-level planning from low-level execution. HiPER factorizes the policy into a high-level planner that proposes subgoals and a low-level executor that carries them out over multiple action steps. To align optimization with this structure, we introduce a key technique called hierarchical advantage estimation (HAE), which carefully assigns credit at both the planning and execution levels. By aggregating returns over the execution of each subgoal and coordinating updates across the two levels, HAE provides an unbiased gradient estimator and provably reduces variance compared to flat generalized advantage estimation. Empirically, HiPER achieves state-of-the-art performance on challenging interactive benchmarks, reaching 97.4\% success on ALFWorld and 83.3\% on WebShop with Qwen2.5-7B-Instruct (+6.6\% and +8.3\% over the best prior method), with especially large gains on long-horizon tasks requiring multiple dependent subtasks. These results highlight the importance of explicit hierarchical decomposition for scalable RL training of multi-turn LLM agents.