Time-Aware Attention for Enhanced Electronic Health Records Modeling

TL;DR

This paper introduces TALE-EHR, a Transformer-based model with a novel time-aware attention mechanism and semantic embeddings, significantly improving the modeling of irregularly timed clinical events in electronic health records for better patient outcome predictions.

Contribution

It presents a new time-aware attention mechanism for Transformers and integrates semantic embeddings from a pre-trained LLM to enhance EHR modeling.

Findings

Outperforms state-of-the-art baselines on disease progression tasks

Effectively models irregular time intervals between clinical events

Demonstrates improved predictive accuracy on MIMIC-IV and PIC datasets

Abstract

Electronic Health Records (EHR) contain valuable clinical information for predicting patient outcomes and guiding healthcare decisions. However, effectively modeling Electronic Health Records (EHRs) requires addressing data heterogeneity and complex temporal patterns. Standard approaches often struggle with irregular time intervals between clinical events. We propose TALE-EHR, a Transformer-based framework featuring a novel time-aware attention mechanism that explicitly models continuous temporal gaps to capture fine-grained sequence dynamics. To complement this temporal modeling with robust semantics, TALE-EHR leverages embeddings derived from standardized code descriptions using a pre-trained Large Language Model (LLM), providing a strong foundation for understanding clinical concepts. Experiments on the MIMIC-IV and PIC dataset demonstrate that our approach outperforms state-of-the-art baselines on tasks such as disease progression forecasting. TALE-EHR underscores the benefit of integrating explicit, continuous temporal modeling with strong semantic representations provides a powerful solution for advancing EHR analysis.