CKD-EHR:Clinical Knowledge Distillation for Electronic Health Records

TL;DR

This paper introduces CKD-EHR, a knowledge distillation framework that enhances disease prediction accuracy and efficiency in EHRs by transferring knowledge from a large language model to a lightweight model, improving clinical diagnosis.

Contribution

The study presents a novel knowledge distillation approach using multi-granularity attention to improve EHR-based disease prediction models, addressing medical knowledge representation and deployment efficiency.

Findings

9% increase in diagnostic accuracy

27% improvement in F1-score

22.2x faster inference speed

Abstract

Electronic Health Records (EHR)-based disease prediction models have demonstrated significant clinical value in promoting precision medicine and enabling early intervention. However, existing large language models face two major challenges: insufficient representation of medical knowledge and low efficiency in clinical deployment. To address these challenges, this study proposes the CKD-EHR (Clinical Knowledge Distillation for EHR) framework, which achieves efficient and accurate disease risk prediction through knowledge distillation techniques. Specifically, the large language model Qwen2.5-7B is first fine-tuned on medical knowledge-enhanced data to serve as the teacher model.It then generates interpretable soft labels through a multi-granularity attention distillation mechanism. Finally, the distilled knowledge is transferred to a lightweight BERT student model. Experimental results show that on the MIMIC-III dataset, CKD-EHR significantly outperforms the baseline model:diagnostic accuracy is increased by 9%, F1-score is improved by 27%, and a 22.2 times inference speedup is achieved. This innovative solution not only greatly improves resource utilization efficiency but also significantly enhances the accuracy and timeliness of diagnosis, providing a practical technical approach for resource optimization in clinical settings. The code and data for this research are available athttps://github.com/209506702/CKD_EHR.