Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning

TL;DR

This paper introduces process reward models (PRMs) for large language models, which provide step-level feedback based on progress, leading to improved reasoning, exploration, and efficiency compared to outcome reward models (ORMs).

Contribution

It proposes a novel method for designing process rewards based on progress measurement under a distinct prover policy, with theoretical characterization and empirical validation showing significant improvements.

Findings

Test-time search with PAVs is >8% more accurate.

Online RL with PAVs yields 5-6x sample efficiency gains.

PAVs are 1.5-5x more compute-efficient than ORMs.

Abstract

A promising approach for improving reasoning in large language models is to use process reward models (PRMs). PRMs provide feedback at each step of a multi-step reasoning trace, potentially improving credit assignment over outcome reward models (ORMs) that only provide feedback at the final step. However, collecting dense, per-step human labels is not scalable, and training PRMs from automatically-labeled data has thus far led to limited gains. To improve a base policy by running search against a PRM or using it as dense rewards for reinforcement learning (RL), we ask: "How should we design process rewards?". Our key insight is that, to be effective, the process reward for a step should measure progress: a change in the likelihood of producing a correct response in the future, before and after taking the step, corresponding to the notion of step-level advantages in RL. Crucially, this progress should be measured under a prover policy distinct from the base policy. We theoretically characterize the set of good provers and our results show that optimizing process rewards from such provers improves exploration during test-time search and online RL. In fact, our characterization shows that weak prover policies can substantially improve a stronger base policy, which we also observe empirically. We validate our claims by training process advantage verifiers (PAVs) to predict progress under such provers, and show that compared to ORMs, test-time search against PAVs is $>8\%$ more accurate, and $1.5-5\times$ more compute-efficient. Online RL with dense rewards from PAVs enables one of the first results with $5-6\times$ gain in sample efficiency, and $>6\%$ gain in accuracy, over ORMs.