Cascading Modular Network (CAM-Net) for Multimodal Image Synthesis

TL;DR

CAM-Net is a versatile deep generative architecture that addresses mode collapse in multimodal image synthesis, producing diverse, high-quality images with fine details across various tasks.

Contribution

It introduces CAM-Net, a unified model based on IMLE that improves detail generation and reduces FID scores across multiple multimodal image synthesis tasks.

Findings

Reduces FID by up to 45.3% compared to baseline

Generates diverse outputs with high-frequency details

Applicable to a broad range of tasks

Abstract

Deep generative models such as GANs have driven impressive advances in conditional image synthesis in recent years. A persistent challenge has been to generate diverse versions of output images from the same input image, due to the problem of mode collapse: because only one ground truth output image is given per input image, only one mode of the conditional distribution is modelled. In this paper, we focus on this problem of multimodal conditional image synthesis and build on the recently proposed technique of Implicit Maximum Likelihood Estimation (IMLE). Prior IMLE-based methods required different architectures for different tasks, which limit their applicability, and were lacking in fine details in the generated images. We propose CAM-Net, a unified architecture that can be applied to a broad range of tasks. Additionally, it is capable of generating convincing high frequency details, achieving a reduction of the Frechet Inception Distance (FID) by up to 45.3% compared to the baseline.