MAPLE: Elevating Medical Reasoning from Statistical Consensus to Process-Led Alignment

TL;DR

This paper introduces a new training paradigm for medical large language models that replaces heuristic majority voting with expert-aligned rewards, significantly improving reasoning accuracy in complex medical scenarios.

Contribution

It proposes integrating medical process reward models with test-time reinforcement learning to guide models with medical correctness rather than consensus.

Findings

Outperforms existing TTRL and PRM methods on four benchmarks.

Demonstrates the importance of structured, step-wise rewards for medical AI.

Enhances model reliability and scalability in medical reasoning.

Abstract

Recent advances in medical large language models have explored Test-Time Reinforcement Learning (TTRL) to enhance reasoning. However, standard TTRL often relies on majority voting (MV) as a heuristic supervision signal, which can be unreliable in complex medical scenarios where the most frequent reasoning path is not necessarily the clinically correct one. In this work, we propose a novel and unified training paradigm that integrates medical process reward models with TTRL to bridge the gap between test-time scaling (TTS) and parametric model optimization. Specifically, we advance the TTRL framework by replacing the conventional MV with a fine-grained, expert-aligned supervision paradigm using Med-RPM. This integration ensures that reinforcement learning is guided by medical correctness rather than mere consensus, effectively distilling search-based intelligence into the model's parametric memory. Extensive evaluations on four different benchmarks have demonstrated that our developed method consistently and significantly outperforms current TTRL and standalone PRM selection. Our findings establish that transitioning from stochastic heuristics to structured, step-wise rewards is essential for developing reliable and scalable medical AI systems