Unified Cross-Modal Medical Image Synthesis with Hierarchical Mixture of Product-of-Experts

TL;DR

This paper introduces MMHVAE, a hierarchical variational auto-encoder model that effectively synthesizes missing medical images across different modalities, addressing challenges in high-resolution generation, missing data estimation, multimodal fusion, and dataset-level learning.

Contribution

The paper presents a novel deep mixture model, MMHVAE, for cross-modal medical image synthesis that handles incomplete data and learns complex multimodal representations.

Findings

Successfully synthesizes missing brain images in MRI and ultrasound modalities.

Outperforms existing methods in image quality and data completion tasks.

Demonstrates robustness to incomplete datasets during training.

Abstract

We propose a deep mixture of multimodal hierarchical variational auto-encoders called MMHVAE that synthesizes missing images from observed images in different modalities. MMHVAE's design focuses on tackling four challenges: (i) creating a complex latent representation of multimodal data to generate high-resolution images; (ii) encouraging the variational distributions to estimate the missing information needed for cross-modal image synthesis; (iii) learning to fuse multimodal information in the context of missing data; (iv) leveraging dataset-level information to handle incomplete data sets at training time. Extensive experiments are performed on the challenging problem of pre-operative brain multi-parametric magnetic resonance and intra-operative ultrasound imaging.