Segment Policy Optimization: Effective Segment-Level Credit Assignment in RL for Large Language Models

TL;DR

This paper introduces Segment Policy Optimization (SPO), a reinforcement learning framework that improves large language model reasoning by using segment-level advantage estimation, balancing granularity and estimation accuracy without requiring a critic model.

Contribution

SPO is a novel RL method that employs segment-level advantage estimation with new strategies for partitioning and advantage calculation, enhancing reasoning accuracy in language models.

Findings

SPO outperforms PPO and GRPO on GSM8K with 6-12% accuracy improvements.

SPO significantly reduces Monte Carlo estimation costs for long chain-of-thought reasoning.

SPO achieves 7-11% improvements over GRPO on MATH500 datasets.

Abstract

Enhancing the reasoning capabilities of large language models effectively using reinforcement learning (RL) remains a crucial challenge. Existing approaches primarily adopt two contrasting advantage estimation granularities: token-level methods (e.g., PPO) aim to provide fine-grained advantage signals but suffer from inaccurate estimation due to difficulties in training an accurate critic model. On the other extreme, trajectory-level methods (e.g., GRPO) solely rely on a coarse-grained advantage signal from the final reward, leading to imprecise credit assignment. To address these limitations, we propose Segment Policy Optimization (SPO), a novel RL framework that leverages segment-level advantage estimation at an intermediate granularity, achieving a better balance by offering more precise credit assignment than trajectory-level methods and requiring fewer estimation points than token-level methods, enabling accurate advantage estimation based on Monte Carlo (MC) without a critic model. SPO features three components with novel strategies: (1) flexible segment partition; (2) accurate segment advantage estimation; and (3) policy optimization using segment advantages, including a novel probability-mask strategy. We further instantiate SPO for two specific scenarios: (1) SPO-chain for short chain-of-thought (CoT), featuring novel cutpoint-based partition and chain-based advantage estimation, achieving $6$-$12$ percentage point improvements in accuracy over PPO and GRPO on GSM8K. (2) SPO-tree for long CoT, featuring novel tree-based advantage estimation, which significantly reduces the cost of MC estimation, achieving $7$-$11$ percentage point improvements over GRPO on MATH500 under 2K and 4K context evaluation. We make our code publicly available at https://github.com/AIFrameResearch/SPO.