Bi-Modality Medical Image Synthesis Using Semi-Supervised Sequential Generative Adversarial Networks

TL;DR

This paper introduces a semi-supervised, sequential GAN framework for bi-modality medical image synthesis that automatically determines synthesis order based on complexity, improving quality and clinical relevance.

Contribution

It proposes a novel sequential GAN with an automatic complexity-based order determination and semi-supervised training for improved bi-modality medical image synthesis.

Findings

Outperforms state-of-the-art methods in visual quality.

Demonstrates clinical significance through user studies.

Effective semi-supervised training reduces overfitting.

Abstract

In this paper, we propose a bi-modality medical image synthesis approach based on sequential generative adversarial network (GAN) and semi-supervised learning. Our approach consists of two generative modules that synthesize images of the two modalities in a sequential order. A method for measuring the synthesis complexity is proposed to automatically determine the synthesis order in our sequential GAN. Images of the modality with a lower complexity are synthesized first, and the counterparts with a higher complexity are generated later. Our sequential GAN is trained end-to-end in a semi-supervised manner. In supervised training, the joint distribution of bi-modality images are learned from real paired images of the two modalities by explicitly minimizing the reconstruction losses between the real and synthetic images. To avoid overfitting limited training images, in unsupervised training, the marginal distribution of each modality is learned based on unpaired images by minimizing the Wasserstein distance between the distributions of real and fake images. We comprehensively evaluate the proposed model using two synthesis tasks based on three types of evaluate metrics and user studies. Visual and quantitative results demonstrate the superiority of our method to the state-of-the-art methods, and reasonable visual quality and clinical significance. Code is made publicly available at https://github.com/hustlinyi/Multimodal-Medical-Image-Synthesis.