Improving Hospital Risk Prediction with Knowledge-Augmented Multimodal EHR Modeling

TL;DR

This paper presents a novel multimodal EHR modeling framework that combines large language models, external knowledge retrieval, and structured data to improve clinical risk prediction accuracy.

Contribution

It introduces a two-stage architecture integrating LLMs and graph-based knowledge retrieval with structured data for enhanced prediction performance.

Findings

Achieved AUC of 0.84 for 30-day readmission

Achieved AUC of 0.92 for in-hospital mortality

Outperformed existing baselines and clinical risk scores

Abstract

Accurate prediction of clinical outcomes using Electronic Health Records (EHRs) is critical for early intervention, efficient resource allocation, and improved patient care. EHRs contain multimodal data, including both structured data and unstructured clinical notes that provide rich, context-specific information. In this work, we introduce a unified framework that seamlessly integrates these diverse modalities, leveraging all relevant available information through a two-stage architecture for clinical risk prediction. In the first stage, a fine-tuned Large Language Model (LLM) extracts crucial, task-relevant information from clinical notes, which is enhanced by graph-based retrieval of external domain knowledge from sources such as a medical corpus like PubMed, grounding the LLM's understanding. The second stage combines both unstructured representations and features derived from the structured data to generate the final predictions. This approach supports a wide range of clinical tasks. Here, we demonstrate its effectiveness on 30-day readmission and in-hospital mortality prediction. Experimental results show that our framework achieves strong performance, with AUC scores of $0.84$ and $0.92$, respectively, despite these tasks involving severely imbalanced datasets, with positive rates ranging from approximately $4\%$ to $13\%$. Moreover, it outperforms all existing baselines and clinical practices, including established risk scoring systems. To the best of our knowledge, this is one of the first frameworks for healthcare prediction which enhances the power of an LLM-based graph-guided knowledge retrieval method by combining it with structured data for improved clinical outcome prediction.