Save the Good Prefix: Precise Error Penalization via Process-Supervised RL to Enhance LLM Reasoning

TL;DR

This paper introduces VPPO, a reinforcement learning method that uses process reward models to better credit correct reasoning steps in large language models, leading to improved reasoning performance.

Contribution

The paper proposes VPPO, a novel RL approach that localizes first errors in reasoning trajectories using PRMs, enhancing credit assignment and model reasoning capabilities.

Findings

VPPO outperforms sparse-reward RL on multiple benchmarks.

VPPO achieves higher Pass@1 and Pass@K scores.

The method provides stable and interpretable learning signals.

Abstract

Reinforcement learning (RL) has emerged as a powerful framework for improving the reasoning capabilities of large language models (LLMs). However, most existing RL approaches rely on sparse outcome rewards, which fail to credit correct intermediate steps in partially successful solutions. Process reward models (PRMs) offer fine-grained step-level supervision, but their scores are often noisy and difficult to evaluate. As a result, recent PRM benchmarks focus on a more objective capability: detecting the first incorrect step in a reasoning path. However, this evaluation target is misaligned with how PRMs are typically used in RL, where their step-wise scores are treated as raw rewards to maximize. To bridge this gap, we propose Verifiable Prefix Policy Optimization (VPPO), which uses PRMs only to localize the first error during RL. Given an incorrect rollout, VPPO partitions the trajectory into a verified correct prefix and an erroneous suffix based on the first error, rewarding the former while applying targeted penalties only after the detected mistake. This design yields stable, interpretable learning signals and improves credit assignment. Across multiple reasoning benchmarks, VPPO consistently outperforms sparse-reward RL and prior PRM-guided baselines on both Pass@1 and Pass@K.