Multi-Modal Perceiver Language Model for Outcome Prediction in Emergency Department

TL;DR

This paper develops a multi-modal language model based on Perceiver for emergency department outcome prediction, integrating text and vital signs to improve diagnosis accuracy and provide insights into multi-modal contributions.

Contribution

It adapts the Perceiver model for healthcare by modifying position encoding for tabular data and demonstrates improved prediction performance using combined modalities.

Findings

Multi-modality improves diagnosis code prediction accuracy.

Vital signs add predictive power for certain disease categories.

Cross-attention analysis reveals how modalities influence predictions.

Abstract

Language modeling have shown impressive progress in generating compelling text with good accuracy and high semantic coherence. An interesting research direction is to augment these powerful models for specific applications using contextual information. In this work, we explore multi-modal language modeling for healthcare applications. We are interested in outcome prediction and patient triage in hospital emergency department based on text information in chief complaints and vital signs recorded at triage. We adapt Perceiver - a modality-agnostic transformer-based model that has shown promising results in several applications. Since vital-sign modality is represented in tabular format, we modified Perceiver position encoding to ensure permutation invariance. We evaluated the multi-modal language model for the task of diagnosis code prediction using MIMIC-IV ED dataset on 120K visits. In the experimental analysis, we show that mutli-modality improves the prediction performance compared with models trained solely on text or vital signs. We identified disease categories for which multi-modality leads to performance improvement and show that for these categories, vital signs have added predictive power. By analyzing the cross-attention layer, we show how multi-modality contributes to model predictions. This work gives interesting insights on the development of multi-modal language models for healthcare applications.