Scaling Medical Reasoning Verification via Tool-Integrated Reinforcement Learning

TL;DR

This paper introduces thod, a reinforcement learning framework that enhances medical reasoning verification by enabling iterative, tool-augmented evidence retrieval, significantly improving accuracy and reducing sampling costs.

Contribution

It presents a novel agentic framework that trains medical reasoning verifiers to iteratively query external medical data during evaluation, addressing limitations of existing reward models.

Findings

Improves MedQA accuracy by 23.5%

Enhances MedXpertQA accuracy by 32.0%

Reduces sampling budget by 8 times

Abstract

Large language models have achieved strong performance on medical reasoning benchmarks, yet their deployment in clinical settings demands rigorous verification to ensure factual accuracy. While reward models offer a scalable approach for reasoning trace verification, existing methods face two limitations: they produce only scalar reward values without explicit justification, and they rely on single-pass retrieval that precludes adaptive knowledge access as verification unfolds. We introduce $\method$, an agentic framework that addresses these limitations by training medical reasoning verifiers to iteratively query external medical corpora during evaluation. Our approach combines tool-augmented verification with an iterative reinforcement learning paradigm that requires only trace-level supervision, alongside an adaptive curriculum mechanism that dynamically adjusts training data distribution. Across four medical reasoning benchmarks, $\method$ achieves substantial gains over existing methods, improving MedQA accuracy by 23.5% and MedXpertQA by 32.0% relative to the base generator in particular. Crucially, $\method$ demonstrates an $\mathbf{8\times}$ reduction in sampling budget requirement compared to prior reward model baselines. These findings establish that grounding verification in dynamically retrieved evidence offers a principled path toward more reliable medical reasoning systems.