MaxCorrMGNN: A Multi-Graph Neural Network Framework for Generalized Multimodal Fusion of Medical Data for Outcome Prediction

TL;DR

MaxCorrMGNN introduces a novel multi-layered graph neural network framework that effectively fuses multimodal medical data by modeling complex non-linear correlations, leading to improved outcome prediction performance.

Contribution

The paper presents the first generalized multi-layered graph neural network for multimodal medical data fusion, leveraging MaxCorr embeddings for non-linear correlation modeling.

Findings

Outperforms state-of-the-art fusion methods on TB dataset

Effectively models complex modality interactions

Enhances outcome prediction accuracy

Abstract

With the emergence of multimodal electronic health records, the evidence for an outcome may be captured across multiple modalities ranging from clinical to imaging and genomic data. Predicting outcomes effectively requires fusion frameworks capable of modeling fine-grained and multi-faceted complex interactions between modality features within and across patients. We develop an innovative fusion approach called MaxCorr MGNN that models non-linear modality correlations within and across patients through Hirschfeld-Gebelein-Renyi maximal correlation (MaxCorr) embeddings, resulting in a multi-layered graph that preserves the identities of the modalities and patients. We then design, for the first time, a generalized multi-layered graph neural network (MGNN) for task-informed reasoning in multi-layered graphs, that learns the parameters defining patient-modality graph connectivity and message passing in an end-to-end fashion. We evaluate our model an outcome prediction task on a Tuberculosis (TB) dataset consistently outperforming several state-of-the-art neural, graph-based and traditional fusion techniques.