Modality Completion via Gaussian Process Prior Variational Autoencoders for Multi-Modal Glioma Segmentation

TL;DR

This paper introduces MGP-VAE, a novel model that imputes missing MRI sub-modalities using Gaussian Process priors, improving multi-modal glioma segmentation when some data are absent.

Contribution

The paper presents MGP-VAE, a new approach leveraging Gaussian Process priors within a VAE to effectively impute missing MRI modalities for better segmentation.

Findings

MGP-VAE outperforms baseline methods on BraTS'19 dataset.

Effective in scenarios with two or three missing sub-modalities.

Improves segmentation accuracy with incomplete multi-modal MRI data.

Abstract

In large studies involving multi protocol Magnetic Resonance Imaging (MRI), it can occur to miss one or more sub-modalities for a given patient owing to poor quality (e.g. imaging artifacts), failed acquisitions, or hallway interrupted imaging examinations. In some cases, certain protocols are unavailable due to limited scan time or to retrospectively harmonise the imaging protocols of two independent studies. Missing image modalities pose a challenge to segmentation frameworks as complementary information contributed by the missing scans is then lost. In this paper, we propose a novel model, Multi-modal Gaussian Process Prior Variational Autoencoder (MGP-VAE), to impute one or more missing sub-modalities for a patient scan. MGP-VAE can leverage the Gaussian Process (GP) prior on the Variational Autoencoder (VAE) to utilize the subjects/patients and sub-modalities correlations. Instead of designing one network for each possible subset of present sub-modalities or using frameworks to mix feature maps, missing data can be generated from a single model based on all the available samples. We show the applicability of MGP-VAE on brain tumor segmentation where either, two, or three of four sub-modalities may be missing. Our experiments against competitive segmentation baselines with missing sub-modality on BraTS'19 dataset indicate the effectiveness of the MGP-VAE model for segmentation tasks.