Toward Better EHR Reasoning in LLMs: Reinforcement Learning with Expert Attention Guidance

TL;DR

This paper introduces EAG-RL, a reinforcement learning framework that enhances LLMs' reasoning in EHR tasks by guiding attention with expert models, leading to better clinical prediction performance.

Contribution

EAG-RL is a novel two-stage training method that intrinsically improves LLM reasoning in EHR analysis through expert-guided trajectories and attention alignment.

Findings

EAG-RL improves LLM EHR reasoning by 14.62% on average.

Enhances robustness to feature perturbations.

Improves generalization to unseen clinical domains.

Abstract

Improving large language models (LLMs) for electronic health record (EHR) reasoning is essential for enabling accurate and generalizable clinical predictions. While LLMs excel at medical text understanding, they underperform on EHR-based prediction tasks due to challenges in modeling temporally structured, high-dimensional data. Existing approaches often rely on hybrid paradigms, where LLMs serve merely as frozen prior retrievers while downstream deep learning (DL) models handle prediction, failing to improve the LLM's intrinsic reasoning capacity and inheriting the generalization limitations of DL models. To this end, we propose EAG-RL, a novel two-stage training framework designed to intrinsically enhance LLMs' EHR reasoning ability through expert attention guidance, where expert EHR models refer to task-specific DL models trained on EHR data. Concretely, EAG-RL first constructs high-quality, stepwise reasoning trajectories using expert-guided Monte Carlo Tree Search to effectively initialize the LLM's policy. Then, EAG-RL further optimizes the policy via reinforcement learning by aligning the LLM's attention with clinically salient features identified by expert EHR models. Extensive experiments on two real-world EHR datasets show that EAG-RL improves the intrinsic EHR reasoning ability of LLMs by an average of 14.62%, while also enhancing robustness to feature perturbations and generalization to unseen clinical domains. These results demonstrate the practical potential of EAG-RL for real-world deployment in clinical prediction tasks. Our code have been available at https://github.com/devilran6/EAG-RL.