A$^2$TGPO: Agentic Turn-Group Policy Optimization with Adaptive Turn-level Clipping

TL;DR

This paper introduces A$^2$TGPO, a novel reinforcement learning method for large language models that adaptively normalizes, accumulates, and clips turn-level signals to improve multi-turn interaction training.

Contribution

It proposes a new approach to leverage intrinsic information gain signals with adaptive normalization, accumulation, and clipping, addressing systematic challenges in RL training of agentic LLMs.

Findings

Normalized IG within turn groups improves stability.

Variance-rescaled accumulation maintains consistent advantage magnitudes.

Adaptive clipping enhances policy updates based on turn informativeness.

Abstract

Reinforcement learning for agentic large language models (LLMs) typically relies on a sparse, trajectory-level outcome reward, making it difficult to evaluate the contribution of individual tool-calls within multi-turn interactions. Existing approaches to such process credit assignment either depend on separate external process reward models that introduce additional consumption, or tree-based structural rollout that merely redistributes the outcome signal while constraining trajectory diversity. A promising alternative leverages the per-turn change in the policy's predicted probability of the ground-truth, termed Information Gain (IG), as an intrinsic process signal without an external evaluator. However, prior work on leveraging IG signals within the RL training loop faces three systematic challenges: normalizing across turns that face heterogeneous positional contexts can distort the relative standing of individual turns, accumulating a variable number of terms causes advantage magnitudes to drift with trajectory depth, and a fixed clipping range governs policy updates identically for turns with vastly different IG signals. In this paper, we propose A$^2$TGPO (Agentic Turn-Group Policy Optimization with Adaptive Turn-level Clipping), which retains IG as the intrinsic signal but re-designs how it is normalized, accumulated, and consumed: (i) turn-group normalization: normalizes IG within each (prompt, turn-index) group so that each turn is compared only against peers at the same interaction depth; (ii) variance-rescaled discounted accumulation: divides cumulative normalized IG by square root of accumulated terms to keep advantage magnitudes comparable across turn positions; and (iii) adaptive turn-level clipping: modulates each turn's clipping range based on its normalized IG, widening the update region for informative turns and narrowing it for uninformative ones.