PRMBench: A Fine-grained and Challenging Benchmark for Process-Level Reward Models

TL;DR

PRMBench is a comprehensive benchmark designed to evaluate the fine-grained error detection capabilities of Process-level Reward Models in complex reasoning tasks, addressing limitations of existing step correctness assessments.

Contribution

The paper introduces PRMBench, a new process-level benchmark with detailed labels, to systematically evaluate PRMs' nuanced error detection abilities, highlighting current weaknesses and guiding future research.

Findings

Current PRMs show significant weaknesses in error detection.

PRMBench reveals challenges in process-level evaluation.

Models vary widely across different error detection dimensions.

Abstract

Process-level Reward Models (PRMs) are crucial for complex reasoning and decision-making tasks, where each intermediate step plays an important role in the reasoning process. Since language models are prone to various types of errors during the reasoning process, PRMs are required to possess nuanced capabilities for detecting various implicit error types in real-world scenarios. However, current benchmarks primarily focus on step correctness, failing to evaluate PRMs' performance systematically. To address this gap, we introduce PRMBench, a process-level benchmark specifically designed to assess the fine-grained error detection capabilities of PRMs. PRMBench comprises 6,216 carefully designed problems and 83,456 step-level labels, evaluating models across multiple dimensions, including simplicity, soundness, and sensitivity. In our experiments on 15 models, spanning both open-source PRMs and closed-source large language models prompted as critic models, we uncover significant weaknesses in current PRMs. These findings underscore the challenges inherent in process-level evaluation and highlight key directions for future research. We hope PRMBench can be a robust bench for advancing research on PRM evaluation and development.