Process Reward Model with Q-Value Rankings

TL;DR

This paper introduces the Process Q-value Model (PQM), a new framework for process reward modeling that uses Q-value rankings and a comparative loss function to better capture decision interdependencies, outperforming traditional classification methods.

Contribution

The paper presents PQM, a novel Markov Decision Process-based approach for process reward modeling that improves reward distribution and decision interdependency understanding.

Findings

PQM outperforms classification-based PRMs across various benchmarks.

The comparative loss function enhances the model's ability to capture decision dynamics.

Ablation studies confirm the effectiveness of the proposed approach.

Abstract

Process Reward Modeling (PRM) is critical for complex reasoning and decision-making tasks where the accuracy of intermediate steps significantly influences the overall outcome. Existing PRM approaches, primarily framed as classification problems, employ cross-entropy loss to independently evaluate each step's correctness. This method can lead to suboptimal reward distribution and does not adequately address the interdependencies among steps. To address these limitations, we introduce the Process Q-value Model (PQM), a novel framework that redefines PRM in the context of a Markov Decision Process. PQM optimizes Q-value rankings based on a novel comparative loss function, enhancing the model's ability to capture the intricate dynamics among sequential decisions. This approach provides a more granular and theoretically grounded methodology for process rewards. Our extensive empirical evaluations across various sampling policies, language model backbones, and multi-step reasoning benchmarks show that PQM outperforms classification-based PRMs. The effectiveness of the comparative loss function is highlighted in our comprehensive ablation studies, confirming PQM's practical efficacy and theoretical advantage.