Graph-Text Multi-Modal Pre-training for Medical Representation Learning

TL;DR

MedGTX is a novel multi-modal pre-trained model that effectively learns joint representations of structured and unstructured EHR data using graph and text encoders, improving performance on clinical tasks.

Contribution

This work introduces MedGTX, a pre-trained model with a graph encoder and cross-modal learning for EHR data, addressing the challenge of combining structured and unstructured information.

Findings

Consistent improvement on clinical benchmarks

Effective joint representation of EHR modalities

Pre-training benefits downstream tasks

Abstract

As the volume of Electronic Health Records (EHR) sharply grows, there has been emerging interest in learning the representation of EHR for healthcare applications. Representation learning of EHR requires appropriate modeling of the two dominant modalities in EHR: structured data and unstructured text. In this paper, we present MedGTX, a pre-trained model for multi-modal representation learning of the structured and textual EHR data. MedGTX uses a novel graph encoder to exploit the graphical nature of structured EHR data, and a text encoder to handle unstructured text, and a cross-modal encoder to learn a joint representation space. We pre-train our model through four proxy tasks on MIMIC-III, an open-source EHR data, and evaluate our model on two clinical benchmarks and three novel downstream tasks which tackle real-world problems in EHR data. The results consistently show the effectiveness of pre-training the model for joint representation of both structured and unstructured information from EHR. Given the promising performance of MedGTX, we believe this work opens a new door to jointly understanding the two fundamental modalities of EHR data.