Clinical-R1: Empowering Large Language Models for Faithful and Comprehensive Reasoning with Clinical Objective Relative Policy Optimization

TL;DR

This paper introduces CRPO, a multi-objective reinforcement learning method that improves the faithfulness and comprehensiveness of large language models in clinical reasoning, surpassing previous methods in truthfulness and completeness.

Contribution

We propose CRPO, a scalable, multi-objective, verifiable RL approach that aligns LLMs with clinical reasoning principles without human annotations, demonstrated on a 3B model.

Findings

CRPO significantly improves truthfulness and completeness in clinical reasoning tasks.

The 3B Clinical-R1 model trained with CRPO outperforms standard methods on benchmarks.

CRPO maintains accuracy while enhancing faithfulness and comprehensiveness.

Abstract

Recent advances in large language models (LLMs) have shown strong reasoning capabilities through large-scale pretraining and post-training reinforcement learning, demonstrated by DeepSeek-R1. However, current post-training methods, such as Grouped Relative Policy Optimization (GRPO), mainly reward correctness, which is not aligned with the multi-dimensional objectives required in high-stakes fields such as medicine, where reasoning must also be faithful and comprehensive. We introduce Clinical-Objective Relative Policy Optimization (CRPO), a scalable, multi-objective, verifiable reinforcement learning method designed to align LLM post-training with clinical reasoning principles. CRPO integrates rule-based and verifiable reward signals that jointly optimize accuracy, faithfulness, and comprehensiveness without relying on human annotation. To demonstrate its effectiveness, we train Clinical-R1-3B, a 3B-parameter model for clinical reasoning. The experiments on three benchmarks demonstrate that our CRPO substantially improves reasoning on truthfulness and completeness over standard GRPO while maintaining comfortable accuracy enhancements. This framework provides a scalable pathway to align LLM reasoning with clinical objectives, enabling safer and more collaborative AI systems for healthcare while also highlighting the potential of multi-objective, verifiable RL methods in post-training scaling of LLMs for medical domains.