Turn-PPO: Turn-Level Advantage Estimation with PPO for Improved Multi-Turn RL in Agentic LLMs

TL;DR

This paper introduces turn-PPO, a novel turn-level advantage estimation method for PPO, improving multi-turn reinforcement learning in agentic language models by enhancing stability and effectiveness over traditional token-level approaches.

Contribution

The paper proposes turn-PPO, a turn-level advantage estimation strategy that outperforms GRPO and token-level PPO in multi-turn RL tasks for language models.

Findings

turn-PPO shows improved stability and performance in multi-turn RL tasks

turn-PPO outperforms GRPO and token-level PPO on WebShop and Sokoban datasets

turn-PPO effectively handles long-horizon reasoning in multi-turn environments

Abstract

Reinforcement learning (RL) has re-emerged as a natural approach for training interactive LLM agents in real-world environments. However, directly applying the widely used Group Relative Policy Optimization (GRPO) algorithm to multi-turn tasks exposes notable limitations, particularly in scenarios requiring long-horizon reasoning. To address these challenges, we investigate more stable and effective advantage estimation strategies, especially for multi-turn settings. We first explore Proximal Policy Optimization (PPO) as an alternative and find it to be more robust than GRPO. To further enhance PPO in multi-turn scenarios, we introduce turn-PPO, a variant that operates on a turn-level MDP formulation, as opposed to the commonly used token-level MDP. Our results on the WebShop and Sokoban datasets demonstrate the effectiveness of turn-PPO, both with and without long reasoning components.