Improving Medical Predictions by Irregular Multimodal Electronic Health Records Modeling

TL;DR

This paper introduces a novel approach to modeling irregular multimodal electronic health records by incorporating irregularity handling mechanisms into each modality and their fusion, significantly improving medical prediction accuracy.

Contribution

It is the first to thoroughly model irregularity in multimodal EHR data, integrating time series and clinical notes with specialized attention mechanisms for better predictions.

Findings

Achieved 6.5% F1 improvement in time series predictions

Achieved 3.6% F1 improvement in clinical note predictions

Achieved 4.3% F1 improvement in multimodal fusion predictions

Abstract

Health conditions among patients in intensive care units (ICUs) are monitored via electronic health records (EHRs), composed of numerical time series and lengthy clinical note sequences, both taken at irregular time intervals. Dealing with such irregularity in every modality, and integrating irregularity into multimodal representations to improve medical predictions, is a challenging problem. Our method first addresses irregularity in each single modality by (1) modeling irregular time series by dynamically incorporating hand-crafted imputation embeddings into learned interpolation embeddings via a gating mechanism, and (2) casting a series of clinical note representations as multivariate irregular time series and tackling irregularity via a time attention mechanism. We further integrate irregularity in multimodal fusion with an interleaved attention mechanism across temporal steps. To the best of our knowledge, this is the first work to thoroughly model irregularity in multimodalities for improving medical predictions. Our proposed methods for two medical prediction tasks consistently outperforms state-of-the-art (SOTA) baselines in each single modality and multimodal fusion scenarios. Specifically, we observe relative improvements of 6.5\%, 3.6\%, and 4.3\% in F1 for time series, clinical notes, and multimodal fusion, respectively. These results demonstrate the effectiveness of our methods and the importance of considering irregularity in multimodal EHRs.