GAGPO: Generalized Advantage Grouped Policy Optimization

TL;DR

GAGPO is a critic-free reinforcement learning method that improves credit assignment in multi-turn environments by constructing non-parametric value proxies from trajectories, leading to better performance and faster learning.

Contribution

GAGPO introduces a novel, critic-free approach for precise temporal credit assignment in multi-turn RL, using grouped value proxies and advantage normalization.

Findings

GAGPO outperforms strong RL baselines on ALFWorld and WebShop.

It achieves faster early-stage learning and improved interaction efficiency.

GAGPO demonstrates smoother optimization dynamics.

Abstract

Reinforcement learning has become a powerful paradigm for post-training large language model agents, yet credit assignment in multi-turn environments remains a challenge. Agents often receive sparse, trajectory-level rewards only at the end of an episode, making it difficult to determine which intermediate actions contributed to success or failure. As a result, propagating delayed outcomes back to individual decision steps without relying on costly auxiliary value models remains an open problem. We propose Generalized Advantage Grouped Policy Optimization (GAGPO), a critic-free reinforcement learning method for precise, step-aligned temporal credit assignment. GAGPO constructs a non-parametric grouped value proxy from sampled rollouts and uses it to compute TD/GAE-style temporal advantages, recursively propagating outcome supervision backward through time. Combined with group-wise advantage normalization and an action-level importance ratio, GAGPO extracts stable, localized optimization signals directly from multi-turn trajectories. Experiments on ALFWorld and WebShop show that GAGPO outperforms strong reinforcement learning baselines. Further analyses demonstrate faster early-stage learning, improved interaction efficiency, and smoother optimization dynamics, suggesting that GAGPO offers a simple yet effective framework for multi-turn agentic reinforcement learning.