MEGAN: Mixture of Experts of Generative Adversarial Networks for Multimodal Image Generation

TL;DR

MEGAN introduces a mixture of specialized generator networks within GANs, trained end-to-end with gating mechanisms, to improve multimodal image generation quality and diversity.

Contribution

The paper proposes MEGAN, a novel ensemble of generator networks with gating, trained jointly, to better model complex data modalities without manual clustering.

Findings

Achieved MS-SSIM score of 0.2470 on CelebA.

Obtained an inception score of 8.33 on CIFAR-10.

Demonstrated diverse and salient subparts in generated images.

Abstract

Recently, generative adversarial networks (GANs) have shown promising performance in generating realistic images. However, they often struggle in learning complex underlying modalities in a given dataset, resulting in poor-quality generated images. To mitigate this problem, we present a novel approach called mixture of experts GAN (MEGAN), an ensemble approach of multiple generator networks. Each generator network in MEGAN specializes in generating images with a particular subset of modalities, e.g., an image class. Instead of incorporating a separate step of handcrafted clustering of multiple modalities, our proposed model is trained through an end-to-end learning of multiple generators via gating networks, which is responsible for choosing the appropriate generator network for a given condition. We adopt the categorical reparameterization trick for a categorical decision to be made in selecting a generator while maintaining the flow of the gradients. We demonstrate that individual generators learn different and salient subparts of the data and achieve a multiscale structural similarity (MS-SSIM) score of 0.2470 for CelebA and a competitive unsupervised inception score of 8.33 in CIFAR-10.