The Bidirectional Process Reward Model

TL;DR

The paper introduces BiPRM, a bidirectional process reward model that evaluates reasoning steps in both directions to improve the accuracy and robustness of reward assessments in large language models.

Contribution

It proposes a novel bidirectional evaluation paradigm with a simple gating mechanism, significantly enhancing process reward modeling with minimal additional parameters and latency.

Findings

BiPRM outperforms unidirectional models across multiple benchmarks.

Achieves an average 10.6% relative gain in solution quality.

Demonstrates robustness and broad applicability in diverse settings.

Abstract

Process Reward Models (PRMs), which assign fine-grained scores to intermediate reasoning steps within a solution trajectory, have emerged as a promising approach to enhance the reasoning quality of Large Language Models (LLMs). However, most existing PRMs rely on a unidirectional left-to-right (L2R) evaluation scheme, which restricts their utilization of global context. In light of this challenge, we propose a novel bidirectional evaluation paradigm, named Bidirectional Process Reward Model (BiPRM). BiPRM incorporates a parallel right-to-left (R2L) evaluation stream, implemented via prompt reversal, alongside the conventional L2R flow. Then a gating mechanism is introduced to adaptively fuse the reward scores from both streams to yield a holistic quality assessment. Remarkably, compared to the original PRM, BiPRM introduces only a 0.3% parameter increase for the gating module, and the parallel execution of two streams incurs merely 5% inference time latency. Our extensive empirical evaluations spanning diverse benchmarks, LLM backbones, PRM objectives and sampling policies demonstrate that BiPRM consistently surpasses unidirectional baselines, achieving an average relative gain of 10.6% over 54 solution-level configurations and 37.7% in 12 step-level error detection scenarios. Generally, our results highlight the effectiveness, robustness and general applicability of BiPRM, offering a promising new direction for process-based reward modeling.