MedPRMBench: A Fine-grained Benchmark for Process Reward Models in Medical Reasoning

TL;DR

MedPRMBench is the first detailed benchmark for evaluating process reward models in medical reasoning, addressing a critical gap in safety and knowledge assessment for healthcare AI.

Contribution

It introduces a comprehensive, fine-grained medical PRM benchmark with a new severity grading system, and provides baseline results highlighting current model weaknesses.

Findings

Medical PRM baseline achieves 87.1% PRMScore, surpassing baselines.

The benchmark covers 14 error types across three categories with severity levels.

Evaluation reveals significant gaps in current models' medical reasoning error detection.

Abstract

Process-Level Reward Models (PRMs) are essential for guiding complex reasoning in large language models, yet existing PRM benchmarks cover only general domains such as mathematics, failing to address medical reasoning -- which is uniquely characterized by safety criticality, knowledge intensity, and diverse error patterns. Without a reliable medical PRM evaluation framework, we cannot quantify models' error detection capabilities in clinical reasoning, leaving their safety in real-world healthcare applications unverified. We propose MedPRMBench, the first process-level reward model benchmark for the medical domain. Built through a three-phase pipeline based on Clinical Reasoning Blueprints (CRBs), MedPRMBench systematically generates high-quality evaluation data from seven medical QA sources, covering 14 fine-grained error types across three categories (Simplicity, Soundness, and Sensitivity) with the first 4-level severity grading system to quantify clinical impact. The benchmark comprises 6{,}500 questions with 13{,}000 reasoning chains and 113{,}910 step-level labels, plus 6{,}879 questions for training. Our medical PRM baseline achieves an 87.1\% overall PRMScore -- substantially surpassing all baselines -- and serves as a plug-and-play verifier that improves downstream medical QA accuracy by 3.2--6.7 percentage points. Systematic evaluation spanning proprietary frontier models, open-source reasoning models, and medical-specialized models reveals critical weaknesses in current models' medical reasoning error detection capabilities, providing clear directions for future PRM improvement.