Unified Brain MR-Ultrasound Synthesis using Multi-Modal Hierarchical Representations

TL;DR

This paper presents MHVAE, a hierarchical variational auto-encoder that effectively synthesizes missing brain MRI and ultrasound images by probabilistically fusing multi-modal data, outperforming existing methods in image synthesis quality.

Contribution

Introduces a novel hierarchical VAE with probabilistic fusion for multi-modal image synthesis, improving upon prior models in handling incomplete data and generating sharper images.

Findings

Outperforms multi-modal VAEs, conditional GANs, and ResViT in image synthesis quality.

Effectively handles incomplete multi-modal image inputs.

Uses adversarial training to produce sharper images.

Abstract

We introduce MHVAE, a deep hierarchical variational auto-encoder (VAE) that synthesizes missing images from various modalities. Extending multi-modal VAEs with a hierarchical latent structure, we introduce a probabilistic formulation for fusing multi-modal images in a common latent representation while having the flexibility to handle incomplete image sets as input. Moreover, adversarial learning is employed to generate sharper images. Extensive experiments are performed on the challenging problem of joint intra-operative ultrasound (iUS) and Magnetic Resonance (MR) synthesis. Our model outperformed multi-modal VAEs, conditional GANs, and the current state-of-the-art unified method (ResViT) for synthesizing missing images, demonstrating the advantage of using a hierarchical latent representation and a principled probabilistic fusion operation. Our code is publicly available \url{https://github.com/ReubenDo/MHVAE}.