A Multimodal Transformer: Fusing Clinical Notes with Structured EHR Data for Interpretable In-Hospital Mortality Prediction

TL;DR

This paper introduces a multimodal transformer that combines clinical notes and structured EHR data to improve in-hospital mortality prediction, with enhanced interpretability through integrated gradients and feature importance analysis.

Contribution

The study presents a novel multimodal transformer model that fuses clinical notes with structured EHR data and incorporates interpretability methods, advancing predictive accuracy and understanding.

Findings

Model outperforms existing methods in AUCPR, AUCROC, and F1 scores.

Integrated gradients effectively identify important clinical notes words.

Shapley values highlight critical structured EHR features.

Abstract

Deep-learning-based clinical decision support using structured electronic health records (EHR) has been an active research area for predicting risks of mortality and diseases. Meanwhile, large amounts of narrative clinical notes provide complementary information, but are often not integrated into predictive models. In this paper, we provide a novel multimodal transformer to fuse clinical notes and structured EHR data for better prediction of in-hospital mortality. To improve interpretability, we propose an integrated gradients (IG) method to select important words in clinical notes and discover the critical structured EHR features with Shapley values. These important words and clinical features are visualized to assist with interpretation of the prediction outcomes. We also investigate the significance of domain adaptive pretraining and task adaptive fine-tuning on the Clinical BERT, which is used to learn the representations of clinical notes. Experiments demonstrated that our model outperforms other methods (AUCPR: 0.538, AUCROC: 0.877, F1:0.490).