ProMed: Shapley Information Gain Guided Reinforcement Learning for Proactive Medical LLMs

TL;DR

ProMed introduces a reinforcement learning framework with Shapley Information Gain rewards to enable medical LLMs to proactively ask questions, improving diagnostic accuracy and interaction quality in clinical settings.

Contribution

The paper presents ProMed, a novel RL-based approach that guides medical LLMs to ask clinically valuable questions using SIG, enhancing proactive medical questioning capabilities.

Findings

ProMed outperforms state-of-the-art methods by 6.29% on medical benchmarks.

ProMed achieves a 54.45% improvement over reactive models.

ProMed generalizes well to out-of-domain cases.

Abstract

Interactive medical questioning is essential in real-world clinical consultations, where physicians must actively gather information from patients. While medical Large Language Models (LLMs) have shown impressive capabilities in static medical question answering, they predominantly operate under a reactive paradigm: generating answers directly without seeking additional information, which risks incorrect diagnoses in such interactive settings. To address this limitation, we propose ProMed, a reinforcement learning (RL) framework that transitions medical LLMs toward a proactive paradigm, equipping them with the ability to ask clinically valuable questions before decision-making. At the core of ProMed is the Shapley Information Gain (SIG) reward, which quantifies the clinical utility of each question by combining the amount of newly acquired information with its contextual importance, estimated via Shapley values. We integrate SIG into a two-stage training pipeline: (1) SIG-Guided Model Initialization uses Monte Carlo Tree Search (MCTS) to construct high-reward interaction trajectories to supervise the model, and (2) SIG-Augmented Policy Optimization, which integrates SIG and enhances RL with a novel SIG-guided Reward Distribution Mechanism that assigns higher rewards to informative questions for targeted optimization. Extensive experiments on two newly curated partial-information medical benchmarks demonstrate that ProMed significantly outperforms state-of-the-art methods by an average of 6.29% and delivers a 54.45% gain over the reactive paradigm, while also generalizing robustly to out-of-domain cases.