Multi-Domain Image Completion for Random Missing Input Data

TL;DR

This paper introduces a multi-domain image completion method using GANs with disentangled representations to handle missing data across multiple modalities, improving performance in medical and facial image tasks.

Contribution

It proposes a novel GAN-based multi-domain image completion approach with disentangled representations and a unified framework for image completion and segmentation.

Findings

Improved segmentation accuracy across three datasets.

Effective handling of random missing domain data.

Enhanced high-level task performance through shared representations.

Abstract

Multi-domain data are widely leveraged in vision applications taking advantage of complementary information from different modalities, e.g., brain tumor segmentation from multi-parametric magnetic resonance imaging (MRI). However, due to possible data corruption and different imaging protocols, the availability of images for each domain could vary amongst multiple data sources in practice, which makes it challenging to build a universal model with a varied set of input data. To tackle this problem, we propose a general approach to complete the random missing domain(s) data in real applications. Specifically, we develop a novel multi-domain image completion method that utilizes a generative adversarial network (GAN) with a representational disentanglement scheme to extract shared skeleton encoding and separate flesh encoding across multiple domains. We further illustrate that the learned representation in multi-domain image completion could be leveraged for high-level tasks, e.g., segmentation, by introducing a unified framework consisting of image completion and segmentation with a shared content encoder. The experiments demonstrate consistent performance improvement on three datasets for brain tumor segmentation, prostate segmentation, and facial expression image completion respectively.