Segmental Advantage Estimation: Enhancing PPO for Long-Context LLM Training

TL;DR

This paper introduces Segmental Advantage Estimation (SAE), a novel method to improve advantage estimation in PPO for long-context LLM training, leading to more stable and efficient reinforcement learning with sparse rewards.

Contribution

SAE partitions sequences into segments to compute advantage estimates, reducing bias and noise compared to GAE, and enhances PPO performance in long-context LLM training.

Findings

SAE outperforms GAE in final scores and stability.

SAE improves sample efficiency across multiple model sizes.

Higher correlation with ground-truth advantage confirms SAE's accuracy.

Abstract

Training Large Language Models (LLMs) for reasoning tasks is increasingly driven by Reinforcement Learning with Verifiable Rewards (RLVR), where Proximal Policy Optimization (PPO) provides a principled framework for stable policy updates. However, the practical application of PPO is hindered by unreliable advantage estimation in the sparse-reward RLVR regime. This issue arises because the sparse rewards in RLVR lead to inaccurate intermediate value predictions, which in turn introduce significant bias when aggregated at every token by Generalized Advantage Estimation (GAE). To address this, we introduce Segmental Advantage Estimation (SAE), which mitigates the bias that GAE can incur in RLVR. Our key insight is that aggregating $n$-step advantages at every token(as in GAE) is unnecessary and often introduces excessive bias, since individual tokens carry minimal information. Instead, SAE first partitions the generated sequence into coherent sub-segments using low-probability tokens as heuristic boundaries. It then selectively computes variance-reduced advantage estimates only from these information-rich segment transitions, effectively filtering out noise from intermediate tokens. Our experiments demonstrate that SAE achieves superior performance, with marked improvements in final scores, training stability, and sample efficiency. These gains are shown to be consistent across multiple model sizes, and a correlation analysis confirms that our proposed advantage estimator achieves a higher correlation with an approximate ground-truth advantage, justifying its superior performance.