IntelliCare: Improving Healthcare Analysis with Variance-Controlled Patient-Level Knowledge from Large Language Models

TL;DR

IntelliCare enhances healthcare prediction models by integrating variance-controlled, patient-specific external knowledge from LLMs, addressing ambiguity issues and improving accuracy across multiple clinical tasks.

Contribution

It introduces a novel framework that mitigates LLM variance and ambiguity, improving EHR analysis with patient-level knowledge augmentation and hybrid calibration techniques.

Findings

Significant performance improvements on clinical prediction tasks.

Effective mitigation of LLM ambiguity and variance issues.

Enhanced personalized healthcare decision support.

Abstract

While pioneering deep learning methods have made great strides in analyzing electronic health record (EHR) data, they often struggle to fully capture the semantics of diverse medical codes from limited data. The integration of external knowledge from Large Language Models (LLMs) presents a promising avenue for improving healthcare predictions. However, LLM analyses may exhibit significant variance due to ambiguity problems and inconsistency issues, hindering their effective utilization. To address these challenges, we propose IntelliCare, a novel framework that leverages LLMs to provide high-quality patient-level external knowledge and enhance existing EHR models. Concretely, IntelliCare identifies patient cohorts and employs task-relevant statistical information to augment LLM understanding and generation, effectively mitigating the ambiguity problem. Additionally, it refines LLM-derived knowledge through a hybrid approach, generating multiple analyses and calibrating them using both the EHR model and perplexity measures. Experimental evaluations on three clinical prediction tasks across two large-scale EHR datasets demonstrate that IntelliCare delivers significant performance improvements to existing methods, highlighting its potential in advancing personalized healthcare predictions and decision support systems.