Dynamic and Generalizable Process Reward Modeling

TL;DR

This paper introduces DG-PRM, a dynamic reward modeling framework for LLMs that uses a reward tree and Pareto dominance to improve cross-domain generalization and fine-grained reward assessment.

Contribution

We propose DG-PRM, a novel reward modeling approach that dynamically selects reward signals and employs Pareto dominance for better generalization and detailed reward evaluation.

Findings

DG-PRM outperforms existing models on standard benchmarks.

It significantly improves performance on dense reward tasks.

DG-PRM demonstrates strong out-of-distribution adaptability.

Abstract

Process Reward Models (PRMs) are crucial for guiding Large Language Models (LLMs) in complex scenarios by providing dense reward signals. However, existing PRMs primarily rely on heuristic approaches, which struggle with cross-domain generalization. While LLM-as-judge has been proposed to provide generalized rewards, current research has focused mainly on feedback results, overlooking the meaningful guidance embedded within the text. Additionally, static and coarse-grained evaluation criteria struggle to adapt to complex process supervision. To tackle these challenges, we propose Dynamic and Generalizable Process Reward Modeling (DG-PRM), which features a reward tree to capture and store fine-grained, multi-dimensional reward criteria. DG-PRM dynamically selects reward signals for step-wise reward scoring. To handle multifaceted reward signals, we pioneeringly adopt Pareto dominance estimation to identify discriminative positive and negative pairs. Experimental results show that DG-PRM achieves stunning performance on prevailing benchmarks, significantly boosting model performance across tasks with dense rewards. Further analysis reveals that DG-PRM adapts well to out-of-distribution scenarios, demonstrating exceptional generalizability.