Fusing Modalities by Multiplexed Graph Neural Networks for Outcome Prediction in Tuberculosis

TL;DR

This paper introduces a novel multiplexed graph neural network framework for effectively fusing multimodal data to improve outcome prediction in tuberculosis, addressing challenges of incomplete and heterogeneous data.

Contribution

The paper proposes a new multiplexed graph neural network approach that explicitly models relationships between modalities for improved multimodal outcome prediction.

Findings

Outperforms state-of-the-art multimodal fusion methods on TB dataset

Effectively handles incomplete and heterogeneous modality data

Improves accuracy of patient outcome predictions

Abstract

In a complex disease such as tuberculosis, the evidence for the disease and its evolution may be present in multiple modalities such as clinical, genomic, or imaging data. Effective patient-tailored outcome prediction and therapeutic guidance will require fusing evidence from these modalities. Such multimodal fusion is difficult since the evidence for the disease may not be uniform across all modalities, not all modality features may be relevant, or not all modalities may be present for all patients. All these nuances make simple methods of early, late, or intermediate fusion of features inadequate for outcome prediction. In this paper, we present a novel fusion framework using multiplexed graphs and derive a new graph neural network for learning from such graphs. Specifically, the framework allows modalities to be represented through their targeted encodings, and models their relationship explicitly via multiplexed graphs derived from salient features in a combined latent space. We present results that show that our proposed method outperforms state-of-the-art methods of fusing modalities for multi-outcome prediction on a large Tuberculosis (TB) dataset.