P2S: Probabilistic Process Supervision for General-Domain Reasoning Question Answering

TL;DR

This paper introduces Probabilistic Process Supervision (P2S), a self-supervised reinforcement learning framework that enhances reasoning in large language models by providing step-by-step process rewards without human annotations.

Contribution

P2S offers a novel method for fine-grained, process-level supervision in RL for reasoning tasks, improving upon outcome-only reward approaches without needing extra annotations.

Findings

P2S significantly improves performance on reading comprehension benchmarks.

P2S outperforms strong baseline models in medical question answering.

The method effectively addresses reward sparsity in reinforcement learning for reasoning.

Abstract

While reinforcement learning with verifiable rewards (RLVR) has advanced LLM reasoning in structured domains like mathematics and programming, its application to general-domain reasoning tasks remains challenging due to the absence of verifiable reward signals. To this end, methods like Reinforcement Learning with Reference Probability Reward (RLPR) have emerged, leveraging the probability of generating the final answer as a reward signal. However, these outcome-focused approaches neglect crucial step-by-step supervision of the reasoning process itself. To address this gap, we introduce Probabilistic Process Supervision (P2S), a novel self-supervision framework that provides fine-grained process rewards without requiring a separate reward model or human-annotated reasoning steps. During reinforcement learning, P2S synthesizes and filters a high-quality reference reasoning chain (gold-CoT). The core of our method is to calculate a Path Faithfulness Reward (PFR) for each reasoning step, which is derived from the conditional probability of generating the gold-CoT's suffix, given the model's current reasoning prefix. Crucially, this PFR can be flexibly integrated with any outcome-based reward, directly tackling the reward sparsity problem by providing dense guidance. Extensive experiments on reading comprehension and medical Question Answering benchmarks show that P2S significantly outperforms strong baselines.