Do We Need to Verify Step by Step? Rethinking Process Supervision from a Theoretical Perspective

TL;DR

This paper demonstrates that outcome supervision in reinforcement learning is statistically comparable to process supervision under standard conditions, challenging the belief that process supervision is inherently more effective for complex reasoning tasks.

Contribution

The paper provides a theoretical proof that outcome supervision is not statistically harder than process supervision, introducing the Change of Trajectory Measure Lemma and connecting outcome and process supervision.

Findings

Outcome supervision is statistically comparable to process supervision.

The Change of Trajectory Measure Lemma bridges trajectory and step-level distributions.

Performance gaps are likely due to algorithms, not statistical difficulty.

Abstract

As large language models have evolved, it has become crucial to distinguish between process supervision and outcome supervision -- two key reinforcement learning approaches to complex reasoning tasks. While process supervision offers intuitive advantages for long-term credit assignment, the precise relationship between these paradigms has remained an open question. Conventional wisdom suggests that outcome supervision is fundamentally more challenging due to the trajectory-level coverage problem, leading to significant investment in collecting fine-grained process supervision data. In this paper, we take steps towards resolving this debate. Our main theorem shows that, under standard data coverage assumptions, reinforcement learning through outcome supervision is no more statistically difficult than through process supervision, up to polynomial factors in horizon. At the core of this result lies the novel Change of Trajectory Measure Lemma -- a technical tool that bridges return-based trajectory measure and step-level distribution shift. Furthermore, for settings with access to a verifier or a rollout capability, we prove that any policy's advantage function can serve as an optimal process reward model, providing a direct connection between outcome and process supervision. These findings suggest that the empirically observed performance gap -- if any -- between outcome and process supervision likely stems from algorithmic limitations rather than inherent statistical difficulties, potentially transforming how we approach data collection and algorithm design for reinforcement learning.