Learning Missing Modal Electronic Health Records with Unified Multi-modal Data Embedding and Modality-Aware Attention

TL;DR

This paper introduces a novel unified multi-modal embedding and modality-aware attention approach to effectively learn from incomplete electronic health records, improving prediction tasks without relying on imputation.

Contribution

The study proposes UMSE and MAA with SB to handle missing modalities in EHRs, enabling better multi-modal learning without separate imputation modules.

Findings

Outperforms baseline models in mortality prediction

Effective handling of missing modalities in EHRs

Improved prediction accuracy on MIMIC-IV dataset

Abstract

Electronic Health Record (EHR) provides abundant information through various modalities. However, learning multi-modal EHR is currently facing two major challenges, namely, 1) data embedding and 2) cases with missing modality. A lack of shared embedding function across modalities can discard the temporal relationship between different EHR modalities. On the other hand, most EHR studies are limited to relying only on EHR Times-series, and therefore, missing modality in EHR has not been well-explored. Therefore, in this study, we introduce a Unified Multi-modal Set Embedding (UMSE) and Modality-Aware Attention (MAA) with Skip Bottleneck (SB). UMSE treats all EHR modalities without a separate imputation module or error-prone carry-forward, whereas MAA with SB learns missing modal EHR with effective modality-aware attention. Our model outperforms other baseline models in mortality, vasopressor need, and intubation need prediction with the MIMIC-IV dataset.