PRPO: Aligning Process Reward with Outcome Reward in Policy Optimization

TL;DR

PRPO enhances policy optimization for large language models by integrating process-level guidance with outcome rewards, leading to improved reasoning accuracy without requiring a value network.

Contribution

It introduces a critic-free method that combines process reward models with outcome rewards through normalization and distribution alignment.

Findings

PRPO improves accuracy from 61.2% to 64.4% on MATH500.

It achieves this with only eight rollouts and no value network.

Demonstrates efficient fine-grained credit assignment in policy optimization.

Abstract

Policy optimization for large language models often suffers from sparse reward signals in multi-step reasoning tasks. Critic-free methods like GRPO assign a single normalized outcome reward to all tokens, providing limited guidance for intermediate reasoning . While Process Reward Models (PRMs) offer dense feedback, they risk premature collapse when used alone, as early low-reward tokens can drive policies toward truncated outputs. We introduce Process Relative Policy Optimization (PRPO), which combines outcome reliability with process-level guidance in a critic-free framework. PRPO segments reasoning sequences based on semantic clues, normalizes PRM scores into token-level advantages, and aligns their distribution with outcome advantages through location-parameter shift. On MATH500, PRPO improves Qwen2.5-Math-1.5B accuracy from 61.2% to 64.4% over GRPO using only eight rollouts and no value network, demonstrating efficient fine-grained credit assignment within critic-free optimization. Code is available at: https://github.com/SchumiDing/srpocode