Multi-Agent Diverse Generative Adversarial Networks

TL;DR

MAD-GAN introduces a multi-generator GAN architecture that effectively addresses mode collapse by encouraging generators to capture diverse data modes, leading to high-quality, varied image synthesis and improved feature representations.

Contribution

The paper proposes MAD-GAN, a novel multi-agent GAN framework that enforces generator diversity through a discriminator designed to identify the source generator of fake samples.

Findings

MAD-GAN produces diverse, high-quality images in tasks like face generation.

MAD-GAN effectively disentangles multiple data modalities.

MAD-GAN improves unsupervised feature learning performance.

Abstract

We propose MAD-GAN, an intuitive generalization to the Generative Adversarial Networks (GANs) and its conditional variants to address the well known problem of mode collapse. First, MAD-GAN is a multi-agent GAN architecture incorporating multiple generators and one discriminator. Second, to enforce that different generators capture diverse high probability modes, the discriminator of MAD-GAN is designed such that along with finding the real and fake samples, it is also required to identify the generator that generated the given fake sample. Intuitively, to succeed in this task, the discriminator must learn to push different generators towards different identifiable modes. We perform extensive experiments on synthetic and real datasets and compare MAD-GAN with different variants of GAN. We show high quality diverse sample generations for challenging tasks such as image-to-image translation and face generation. In addition, we also show that MAD-GAN is able to disentangle different modalities when trained using highly challenging diverse-class dataset (e.g. dataset with images of forests, icebergs, and bedrooms). In the end, we show its efficacy on the unsupervised feature representation task. In Appendix, we introduce a similarity based competing objective (MAD-GAN-Sim) which encourages different generators to generate diverse samples based on a user defined similarity metric. We show its performance on the image-to-image translation, and also show its effectiveness on the unsupervised feature representation task.

Tables

Table 1. Table 1 : Synthetic experiment on 1D GMM (Fig.  4 ).

GAN Variants	Chi-square$(\times {10}^5)$	KL-Div
DCGAN [26]	0.90	0.322
WGAN [3]	1.32	0.614
BEGAN [5]	1.06	0.944
GoGAN [16]	2.52	0.652
Unrolled GAN [22]	3.98	1.321
Mode-Reg DCGAN [7]	1.02	0.927
InfoGAN [8]	0.83	0.21
MA-GAN	1.39	0.526
MAD-GAN (Our)	0.24	0.145

Table 2. Table 2 : Synthetic experiment with different number of MAD-GAN generators (same setup as in Fig.  4 ).

# Generators	Chi-square $(\times {10}^7)$	KL-Div
1	1.27	0.57
2	1.38	0.42
3	3.15	0.71
4	0.39	0.28
5	3.05	0.88
6	0.54	0.29
7	0.97	0.78
8	4.83	0.68

Table 3. Table 3 : Stacked-MNIST experiments and comparisons. Note that three generators are used for MAD-GAN.

GAN Variants	KL Div	# Modes Covered
DCGAN [26]	2.15	712
WGAN [3]	1.02	868
BEGAN [5]	1.89	819
GoGAN [16]	2.89	672
Unrolled GAN [22]	1.29	842
Mode-Reg DCGAN [7]	1.79	827
InfoGAN [8]	2.75	840
MA-GAN	3.4	700
MAD-GAN (Our)	0.91	890

Table 4. Table 4 : Compositional-MNIST experiments and comparisons. Note that three generators are used for MAD-GAN.

GAN Variants	KL Div	# Modes Covered
DCGAN [26]	0.18	980
WGAN [3]	0.25	1000
BEGAN [5]	0.19	999
GoGAN [16]	0.87	972
Unrolled GAN [22]	0.091	1000
Mode-Reg DCGAN [7]	0.12	992
InfoGAN [8]	0.47	990
MA-GAN	1.62	997
MAD-GAN (Our)	0.074	1000

Table 5. Table 5 : Non-Parametric density estimation architecture for discriminators (Disc), encoders (Enc), decoders (Dec), and Q Network (QNet). nGen is number of generators, fc is fully connected layer.

DCGAN, Unrolled GAN, InfoGAN, MA-GAN Disc	Mode-Reg DCGAN Disc	Mode-Reg DCGAN Enc	WGAN, GoGAN Disc	BEGAN Enc	BEGAN Dec	MAD-GAN Disc	InfoGAN QNet
Input: 1	1	1	32	1	1	1	1
fc: 128, leaky relu
fc: 128, leaky relu
fc: 1	1	64	1	32	1	5 (nGen+1)	5
sigmoid		identity				softmax

Table 6. Table 6 : Generator architecture for 1000 1000 1000 class stacked-MNIST experiment. For MAD-GAN, all the layers except those mentioned in last two rows are shared.

	number outputs	stride
Input: $z\sim 𝒩(0,I_{256})$
Fully connected	4 * 4 * 64
Reshape to image 4,4,64
Transposed Convolution	32	2
Transposed Convolution	16	2
Transposed Convolution	8	2
Convolution	3	1

Table 7. Table 7 : Discriminator architecture for 1000 1000 1000 class stacked-MNIST experiment. For MAD-GAN, with k 𝑘 k generators, it is adapted to have k + 1 𝑘 1 k+1 dimensional last layer output. For InfoGAN, with 156 dimensional salient variables and 100 dimensional incompressible noise, it is adapted to have 156 156 156 dimensional output for Q network.

	number outputs	stride
Input: 32x32 Color Image
Convolution	4	2
Convolution	8	2
Convolution	16	2
Flatten
Fully Connected	1

Table 8. Table 8 : Discriminator architecture for 1000 1000 1000 class compositional-MNIST experiment. For MAD-GAN, with k 𝑘 k generators, it is adapted to have k + 1 𝑘 1 k+1 dimensional last layer output. For InfoGAN, with 156 dimensional salient variables and 100 dimensional incompressible noise, it is adapted to have 156 156 156 dimensional output for Q network.

	number outputs	stride
Input: Color Image (64x64)
Convolution	64	2
Convolution	128	2
Convolution	256	2
Convolution	512	2
Flatten
Fully Connected	1

Table 9. Table 9 : DCGAN Discriminator: It is adapted to have k + 1 𝑘 1 k+1 dimensional last layer output for MAD-GAN with k 𝑘 k generators. (normalizer is softmax).

Discriminator D
Input 64x64 Color Image
4x4 conv. 64 leakyRELU. stride 2. batchnorm
4x4 conv. 128 leakyRELU. stride 2. batchnorm
4x4 conv. 256 leakyRELU. stride 2. batchnorm
4x4 conv. 512 leakyRELU. stride 2. batchnorm
4x4 conv. output leakyRELU. stride 1

Table 10. Table 10 : DCGAN Generator: All the layers except the last one are shared among all the three generators.

Generator G
Input $\in {ℝ}^{100}$
4x4 upconv. 512 RELU.batchnorm.shared
4x4 upconv. 256 RELU. stride 2.batchnorm.shared
4x4 upconv. 128 RELU. stride 2.batchnorm.shared
4x4 upconv. 64 RELU. stride 2.batchnorm.shared
4x4 upconv. 3 tanh. stride 2

Table 11. Table 11 : Discriminator architecture for diverse-class data generation and diverse face generation: The last layer output is k + 1 𝑘 1 k+1 dimensional for MAD-GAN with k 𝑘 k generators (normalizer is softmax). ‘RESIDUAL’ layer is elaborated in Tab.  12 .

Residual Discriminator D
Input 64x64 Color Image
7x7 conv. 64 leakyRELU. stride 2. pad 1. batchnorm
3x3 conv. 64 leakyRELU. stride 2. pad 1. batchnorm
3x3 conv. 128 leakyRELU. stride 2.pad 1. batchnorm
3x3 conv. 256 leakyRELU. stride 2. pad 1. batchnorm
3x3 conv. 512 leakyRELU. stride 2. pad 1. batchnorm
3x3 conv. 512 leakyRELU. stride 2. pad 1. batchnorm
3x3 conv. 512 leakyRELU. stride 2. pad 1. batchnorm
RESIDUAL-(N512, K3, S1, P1)
RESIDUAL-(N512, K3, S1, P1)
RESIDUAL-(N512, K3, S1, P1)

Table 12. Table 12 : Residual layer description for Tab.  11 .

RESIDUAL-Residual Layer
Input: previous-layer-output
c1: CONV-(N512, K3, S1, P1), BN, ReLU
c2: CONV-(N512, K3, S2, P1), BN
SUM(c2,previous-layer-output)

Table 13. Table 13 : The misclassification error of MAD-GAN and MAD-GAN-Sim on SVHN while varying the number of generators.

Technique	2 Gen	3 Gen	4 Gen
MAD-GAN	20.5%	18.2%	17.5%
MAD-GAN-Sim	20.2%	19.6%	18.3%

Equations

$$\underset{{\theta }_g}{\min }\underset{{\theta }_d}{\max }$$

$$\displaystyle+\operatorname{\mathbb{E}}_{z\sim p_{z}}\log\big{(}1-D(G(z;\theta_{g});\theta_{d})\big{)}$$

$$\underset{{\theta }_d}{\max }{E}_{x\sim p}H(\delta ,D(x;{\theta }_d))$$

$${\mathbb{E}}_{x\sim p_d}\log D_{k+1}(x;{\theta }_d)+{\mathbb{E}}_{z\sim p_z}\log (1-D_{k+1}(G_i(z;{\theta }_g^i);{\theta }_d))$$

$$\displaystyle KL\Big{(}p_{d}(x)||p_{avg}(x)\Big{)}+kKL\Big{(}\frac{1}{k}\sum_{i=1}^{k}p_{g_{i}}(x)||p_{avg}(x)\Big{)}$$

$$-(k+1)\log (k+1)+k\log k$$

$${\mathbb{E}}_{x\sim p_d}\log D_{k+1}(x)+\sum _{i=1}^k{\mathbb{E}}_{x\sim p_{g_i}}\log (1-D_{k+1}(x))$$

$$\displaystyle\mathbb{E}_{x\sim p_{d}}\log\Bigg{[}\frac{p_{d}(x)}{p_{d}(x)+\sum_{i=1}^{k}p_{g_{i}}(x)}\Bigg{]}+$$

$$\displaystyle\sum_{i=1}^{k}\mathbb{E}_{x\sim p_{g_{i}}}\log\Bigg{[}\frac{\sum_{i=1}^{k}p_{g_{i}}(x)}{p_{d}(x)+\sum_{i=1}^{k}p_{g_{i}}(x)}\Bigg{]}$$

$$\displaystyle=\mathbb{E}_{x\sim p_{d}}\log\Bigg{[}\frac{p_{d}(x)}{p_{avg}(x)}\Bigg{]}+k\mathbb{E}_{x\sim p_{g}}\log\Bigg{[}\frac{p_{g}(x)}{p_{avg}(x)}\Bigg{]}$$

$$-(k+1)\log (k+1)+k\log k$$

$$D_{k+1}(x)$$

$$D_i(x)$$

$${\mathbb{E}}_{x\sim p_d}\log D_{k+1}(x)+\sum _{i=1}^k{\mathbb{E}}_{x_i\sim p_{g_i}}\log D_i(x_i)$$

$${\int }_x{p}_d(x)\log D_{k+1}(x)dx+\sum _{i=1}^k{\int }_x{p}_{g_i}(x)\log D_i(x)dx$$

$$={\int }_{x\in p}\sum _{i=1}^{k+1}{p}_i(x)\log D_i(x)dx$$

$$\underset{\mathbf{y}}{\max }\sum _{i=1}^n{a}_i\log y_i,s.t.\sum _i^n{y}_i=1$$

$$L(\mathbf{y},\lambda )=\sum _{i=1}^n{a}_i\log y_i+\lambda (\sum _{i=1}^n{y}_i-1)$$

$$\frac{a_i}{y_i}+\lambda =0,\sum _{i=1}^n{y}_i-1=0$$

$$D(G_i(z;{\theta }_g^i);{\theta }_d)\ge D(G_j(z;{\theta }_g^j);{\theta }_d)$$

$$\displaystyle+\Delta\big{(}\phi(g_{i}(z)),\phi(g_{j}(z))\big{)},\;\;\forall j\in K\setminus i$$

$$\displaystyle U(\theta_{g}^{i},\theta_{d}):=f\Big{(}D(G_{i}(z;\theta_{g}^{i});\theta_{d})-$$

$$\displaystyle\frac{1}{k-1}\sum_{j\in K\setminus i}\big{(}D(G_{j}(z;\theta_{g}^{j});\theta_{d})+\Delta(\psi_{i},\psi_{j})\Big{)}$$

$$\underset{{\theta }_g^i}{\min }V({\theta }_d,{\theta }_g^i)-\lambda U({\theta }_g^i,{\theta }_d)$$

Full text

1 Introduction

Generative models have attracted considerable attention recently. The underlying idea behind such models is to attempt to capture the distribution of high-dimensional data such as images and texts. Though these models are highly useful in various applications, it is computationally expensive to train them as they require intractable integration in a very high-dimensional space. This drastically limits their applicability. However, recently there has been considerable progress in deep generative models – conglomerate of deep neural networks and generative models – as they do not explicitly require the intractable integration, and can be efficiently trained using back-propagation algorithm. Two such famous examples are Generative Adversarial Networks (GANs) [13] and Variational Autoencoders [17].

In this paper we focus on GANs as they are known to produce sharp and plausible images. Briefly, GANs employ a generator and a discriminator where both are involved in a minimax game. The task of the discriminator is to learn the difference between real samples (from true data distribution $p_{d}$) and fake samples (from generator distribution $p_{g}$). Whereas, the task of the generator is to maximize the mistakes of the discriminator. At convergence, the generator learns to produce real looking images. A few successful applications of GANs are video generation [30], image inpainting [25], image manipulation [33], 3D object generation [31], interactive image generation using few brush strokes [33], image super-resolution [20], diagrammatic abstract reasoning [18] and conditional GANs [23, 27].

Despite the remarkable success of GAN, it suffers from the major problem of mode collapse [2, 7, 8, 22, 28]. Though, theoretically, convergence guarantees the generator learning the true data distribution. However, practically, reaching the true equilibrium is difficult and not guaranteed, which potentially leads to the aforementioned problem of mode collapse. Broadly speaking, there are two schools of thought to address the issue: (1) improving the learning of GANs to reach better optima [2, 22, 28]; and (2) explicitly enforcing GANs to capture diverse modes [7, 8, 21]. Here we focus on the latter.

Borrowing from the multi-agent algorithm [1] and coupled GAN [21], we propose to use multiple generators with one discriminator. We call this framework the Multi-Agent GAN architecture, as shown in Fig. 2. In detail, similar to the standard GAN, the objective of each generator here is to maximize the mistakes of the common discriminator. Depending on the task, it might be useful for different generators to share information. This is done using the initial layer parameters of generators. Another reason behind sharing these parameters is the fact that initial layers capture low-frequency structures which are almost the same for a particular type of dataset (for example, faces), therefore, sharing them reduces redundant computations. However, when the dataset contains images from completely different modalities, one can avoid sharing these parameters. Naively using multiple generators may lead to the trivial solution where all the generators learn to generate similar samples. To resolve this issue and generate different visually plausible samples capturing diverse high probability modes, we propose to modify the objective function of the discriminator. In the modified objective, along with finding the real and the fake samples, the discriminator also has to correctly identify the generator that generated the given fake sample. Intuitively, in order to succeed in this task, the discriminator must learn to push generations corresponding to different generators towards different identifiable modes. Combining the Multi-Agent GAN architecture with the diversity enforcing term allows us to generate diverse plausible samples, thus the name Multi-Agent Diverse GAN (MAD-GAN).

As an example, an intuitive setting where mode collapse occurs is when a GAN is trained on a dataset containing images from different modalities/classes. For example, a diverse-class dataset containing images such as forests, iceberg, and bedrooms. This is of particular interest as it not only requires the model to disentangle intra-class variations, it also requires inter-class disentanglement. Fig. 1 demonstrates the surprising effectiveness of MAD-GAN in this challenging setting. Generators among themselves are able to disentangle inter-class variations, and each generator is also able to capture intra-class variations.

In addition, we analyze MAD-GAN through extensive experiments and compare it with several variants of GAN. First, for the proof of concept, we perform experiments in controlled settings using synthetic dataset (mixture of Gaussians), and complicated Stacked/Compositional MNIST datasets with hand engineered modes. In these settings, we empirically show that our approach outperforms all other GAN variants we compare with, and is able to generate high quality samples while capturing large number of modes. In a more realistic setting, we show high quality diverse sample generations for the challenging tasks of image-to-image translation [14] (conditional GAN) and face generation [8, 26]. Using the SVHN dataset [24], we also show the efficacy of our framework for learning the feature representation in an unsupervised setting.

We also provide theoretical analysis of this approach and show that the proposed modification in the objective of discriminator allows generators to learn together as a mixture model where each generator represents a mixture component. We show that at convergence, the global optimum value of $-(k+1)\log(k+1)+k\log k$ is achieved, where $k$ is the number of generators.

2 Related Work

The recent work called InfoGAN [8] proposed an information-theoretic extension to GANs in order to address the problem of mode collapse. Briefly, InfoGAN disentangles the latent representation by assuming a factored representation of the latent variables. In order to enforce that the generator learns factor specific generations, InfoGAN maximizes the mutual information between the factored latents and the generator distribution. Che et al. [7] proposed a mode regularized GAN (ModeGAN) which uses an encoder-decoder paradigm. The basic idea behind ModeGAN is that if a sample from the true data distribution $p_{d}$ belongs to a particular mode, then the sample generated by the generator (fake sample) when the true sample is passed through the encoder-decoder is likely to belong to the same mode. ModeGAN assumes that there exists enough true samples from a mode for the generator to be able to capture it. Another work by Metz et al. [22] proposed a surrogate objective for the update of the generator with respect to the unrolled optimization of the discriminator (UnrolledGAN) to address the issue of convergence of the training process of GANs. This improves the training process of the generator which in turn allow the generators to explore better coverage to true data distribution.

Liu et al. [21] presented Coupled GAN, a method for training two generators with shared parameters to learn the joint distribution of the data. The shared parameters guide both the generators towards similar subspaces but since they are trained independently on two domains, they promote diverse generations. Durugkar et al. [10] proposed a model with multiple discriminators whereby an ensemble of multiple discriminators have been shown to stabilize the training of the generator by guiding it to produce better samples.

W-GAN [3] is a recent technique which employs integral probability metrics based on the earth mover distance rather than the JS-divergences that the original GAN uses. BEGAN [5] builds upon W-GAN using an autoencoder based equilibrium enforcing technique alongside the Wasserstein distance. DCGAN [26] was a seminal technique which used a fully convolutional generator and discriminator for the first time along with the introduction of batch normalization thus stabilizing the training procedure, and was able to generate compelling generations. GoGAN [16] introduced a training procedure for the training of the discriminator using a maximum margin formulation alongside the earth mover distance based on the Wasserstein-1 metric. [4] introduced a technique and theoretical formulation stating the importance of multiple generators and discriminators in order to completely model the data distribution. In terms of employing multiple generators, our work is closest to [4, 21, 11]. However, while using multiple generators, our method explicitly enforces them to capture diverse modes.

3 Preliminaries

Here we present a brief review of GANs [13]. Given a set of samples $\mathcal{D}=(x_{i})_{i=1}^{n}$ from the true data distribution $p_{d}$, the GAN learning problem is to obtain the optimal parameters $\theta_{g}$ of a generator $G(z;\theta_{g})$ that can sample from an approximate data distribution $p_{g}$, where $z\sim p_{z}$ is the prior input noise (e.g. samples from a normal distribution). In order to learn the optimal $\theta_{g}$, the GAN objective (Eq. (3)) employs a discriminator $D(x;\theta_{d})$ that learns to differentiate between a real (from $p_{d}$) and a fake (from $p_{g}$) sample $x$. The overall GAN objective is:

[TABLE]

The above objective is optimized in a block-wise manner where $\theta_{d}$ and $\theta_{g}$ are optimized one at a time while fixing the other. For a given sample $x$ (either from $p_{d}$ or $p_{g}$) and the parameter $\theta_{d}$, the function $D(x;\theta_{d})\in[0,1]$ produces a score that represents the probability of $x$ belonging to the true data distribution $p_{d}$ (or probability of it being real). The objective of the discriminator is to learn parameters $\theta_{d}$ that maximizes this score for the true samples (from $p_{d}$) while minimizing it for the fake ones $\tilde{x}=D(z;\theta_{g})$ (from $p_{g}$). In the case of generator, the objective is to minimize \operatorname{\mathbb{E}}_{z\sim p_{z}}\log\big{(}1-D(G(z;\theta_{g});\theta_{d})\big{)}, equivalently maximize $\operatorname{\mathbb{E}}_{z\sim p_{z}}\log D(G(z;\theta_{g});\theta_{d})$. Thus, the generator learns to maximize the scores for the fake samples (from $p_{g}$), which is exactly the opposite to what discriminator is trying to achieve. In this manner, the generator and the discriminator are involved in a minimax game where the task of the generator is to maximize the mistakes of the discriminator. Theoretically, at equilibrium, the generator learns to generate real samples, which means $p_{g}=p_{d}$.

4 Multi-Agent Diverse GAN

In the GAN objective, one can argue that the task of a generator is much harder than that of the discriminator as it has to produce real looking images to maximize the mistakes of the discriminator. This, along with the minimax nature of the objective raise several challenges for GANs [2, 7, 8, 22, 28]: (1) mode collapse; (2) difficult optimization; and (3) trivial solution. In this work we propose a new framework to address the first challenge of mode collapse by increasing the capacity of the generator while using well known tricks to partially avoid other challenges [2].

Briefly, we propose a Multi-Agent GAN architecture that employs multiple generators and one discriminator in order to generate different samples from high probability regions of the true data distribution. In addition, theoretically, we show that our formulation allows generators to act as a mixture model with each generator capturing one component.

4.1 Multi-Agent GAN Architecture

Here we describe our proposed architecture (Fig. 2). It involves $k$ generators and one discriminator. In the case of homogeneous data (all the images belong to same class, e.g. faces or birds), we allow all the generators to share information by tying most of the initial layer parameters. This is essential to avoid redundant computations as initial layers of a generator capture low-frequency structures which are almost the same for a particular type of dataset. This also allows different generators to converge faster. However, in the case of diverse-class data (e.g. dataset with a mixture of different classes such as forests, icebergs etc.), it is necessary to avoid sharing these parameters to allow each generator to capture content specific structures. Thus, the extent to which one should share these parameters depends on the task at hand.

More specifically, given $z\sim p_{z}$ for the $i$-th generator, similar to the standard GAN, the first step involves generating a sample (for example, an image) $\tilde{x}_{i}$. Since each generator receives the same latent input sampled from the same distribution, naively using this simple approach may lead to the trivial solution where all the generators learn to generate similar samples. In what follows, we propose an intuitive solution to avoid this issue and allow the generators to capture diverse modes.

4.2 Enforcing Diverse Modes

Inspired by the discriminator formulation for the semi-supervised learning [28], we use a generator identification based objective function that, along with minimizing the score $D(\tilde{x};\theta_{d})$, requires the discriminator to identify the generator that generated the given fake sample $\tilde{x}$. In order to do so, as opposed to the standard GAN objective function where the discriminator outputs a scalar value, we modify it to output $k+1$ soft-max scores. In more detail, given the set of $k$ generators, the discriminator produces a soft-max probability distribution over $k+1$ classes. The score at $(k+1)$-th index $(D_{k+1}(.))$ represents the probability that the sample belongs to the true data distribution and the score at $j\in\{1,\dots,k\}$-th index represents the probability of it being generated by the $j$-th generator. Under this setting, while learning $\theta_{d}$, we optimize the cross-entropy between the soft-max output of the discriminator and the Dirac delta distribution $\delta\in\{0,1\}^{k+1}$, where for $j\in\{1,\dots,k\}$, $\delta(j)=1$ if the sample belongs to the $j$-th generator, otherwise $\delta(k+1)=1$. Thus, the objective of the discriminator, which is optimizing $\theta_{d}$ while keeping $\theta_{g}$ constant (refer Eq. (3)), is modified to:

[TABLE]

where, $Supp(p)=\cup_{i=1}^{k}Supp(p_{g_{i}})\cup Supp(p_{d})$ and $H(.,.)$ is the negative of the cross entropy function. Intuitively, in order to correctly identify the generator that produced a given fake sample, the discriminator must learn to push different generators towards different identifiable modes. However, the objective of each generator remains the same as in the standard GAN. Thus, for the $i$-th generator, the objective is to minimize the following:

[TABLE]

To update the parameters, the gradient for each generator is simply computed as $\nabla_{\theta_{g}^{i}}\log(1-D_{k+1}(G_{i}(z;\theta_{g}^{i});\theta_{d}))$. Notice that all the generators in this case can be updated in parallel. For the discriminator, given $x\sim p$ (can be real or fake) and corresponding $\delta$, the gradient is $\nabla_{\theta_{d}}\log D_{j}(x;\theta_{d})$, where $D_{j}(x;\theta_{d})$ is the $j$-th index of $D(x;\theta_{d})$ for which $\delta(j)=1$. Therefore, using this approach requires very minor modifications to the standard GAN optimization algorithm and can be easily used with different variants of GAN. An intuitive visualization is shown in Fig. 3.

Theorem 1 shows that the above objective function actually allows generators to form a mixture model where each generator represents a mixture component and the global optimum of $-(k+1)\log(k+1)+k\log k$ is achieved when $p_{d}=\frac{1}{k}\sum_{i=1}^{k}p_{g_{i}}$. Notice that, at $k=1$, which is the case with one generator, we obtain exactly the same Jensen-Shannon divergence based objective function as shown in [13] with the optimal value of $-\log 4$.

Theorem 1**.**

Given the optimal discriminator, the objective for training the generators boils down to minimizing

[TABLE]

where, $p_{avg}(x)=\frac{p_{d}(x)+\sum_{i=1}^{k}p_{g_{i}}(x)}{k+1}$. The above objective function obtains its global minimum if $p_{d}=\frac{1}{k}\sum_{i=1}^{k}p_{g_{i}}$ with the objective value of $-(k+1)\log(k+1)+k\log k$.

Proof.

The joint objective of all the generators is to minimize the following:

[TABLE]

Using Corollary 1, we substitute the optimal discriminator in the above equation and obtain:

[TABLE]

where, $p_{g}=\frac{\sum_{i=1}^{k}p_{g_{i}}}{k}$ and $p_{avg}(x)=\frac{p_{d}(x)+\sum_{i=1}^{k}p_{g_{i}}(x)}{k+1}$. Note that, Eq. (1) is exactly the same as Eq. (1). When $p_{d}=\frac{\sum_{i=1}^{k}p_{g_{i}}}{k}$, both the KL terms become zero and the global minimum is achieved. ∎

Corollary 1**.**

For fixed generators, the optimal distribution learned by the discriminator D has the following form:

[TABLE]

where, $D_{i}(x)$ represents the $i$-th index of $D(x;\theta_{d})$, $p_{d}$ the true data distribution, and $p_{g_{i}}$ the distribution learned by the $i$-th generator.

Proof.

For fixed generators, the objective function of the discriminator is to maximize

[TABLE]

where, $\sum_{i=1}^{k+1}D_{i}(x)=1$ and $D_{i}(x)\in[0,1],\forall i$. The above equation can be written as:

[TABLE]

where, $p_{k+1}(x):=p_{d}(x)$, $p_{i}(x):=p_{g_{i}}(x),\forall i\in\{1,\cdots,k\}$, and $Supp(p)=\bigcup_{i=1}^{k}Supp(p_{g_{i}})\bigcup Supp(p_{d})$, Therefore, for a given $x$, the optimum of objective function defined in Eq. (1) with constraints defined above can be obtained using Proposition 1. ∎

Proposition 1**.**

Given ${\bf y}=(y_{1},\cdots,y_{n})$, $y_{i}\geq 0$, and $a_{i}\in\mathbb{R}$, the optimal solution for the objective function defined below is achieved at $y_{i}^{*}=\frac{a_{i}}{\sum_{i=1}^{n}a_{i}},\forall i$

[TABLE]

Proof.

The Lagrangian of the above problem is:

[TABLE]

Differentiating w.r.t $y_{i}$ and $\lambda$, and equating to zero,

[TABLE]

Solving the above two equations, we obtain $y_{i}^{*}=\frac{a_{i}}{\sum_{i=1}^{n}a_{i}}$. ∎

5 Experiments

We present an extensive quantitative and qualitative analysis of MAD-GAN on various synthetic and real-world datasets. First, we use a simple 1D mixture of Gaussians and also Stacked/Compositional MNIST dataset (1000 modes) to compare MAD-GAN with several known variants of GANs, such as DCGAN [26], WGAN [3], BEGAN [5], GoGAN [16], Unrolled GAN [22], Mode-Reg GAN [7] and InfoGAN [8]. Furthermore, we created another baseline, called MA-GAN (Multi-Agent GAN), which is a trivial extension of GAN with multiple generators and one discriminator. As opposed to MAD-GAN, MA-GAN has a simple Multi-Agent architecture without modifications to the objective of the discriminator. This comparison allows us to understand the effect of explicitly enforcing diversity in the objective of the MAD-GAN. We use KL-divergence [19] and number of modes recovered [7] as the criterion for comparisons and show superior results compared to all the other methods. Additionally, we show diverse generations for the challenging tasks of image-to-image translation [14], diverse-class data generation, and face generation. It is non-trivial to devise a metric to evaluate diversity on these high quality generation tasks, so we perform qualitative assessment. Note that, the image-to-image translation objective is known to learn the delta distribution, thus, it is agnostic to the input noise vector. However, we show that MAD-GAN is able to produce highly plausible diverse generations for this task. In the end, we show the efficacy of MAD-GAN in unsupervised feature representation learning task. We provide detailed overview of the architectures, datasets, and the parameters used in our experiments in the Appendix C.

In the case of InfoGAN [8], we varied the dimension of the categorical variable, depicting the number of modes, to obtain the best cross-validated results.

5.1 Non-Parametric Density Estimation

In order to understand the behavior of MAD-GAN and different state-of-the-art GAN models, we first perform a very simple synthetic experiment, much easier than generating high-dimensional complex images. We consider a distribution of 1D GMM [6] having five mixture components with modes at 10, 20, 60, 80 and 110, and standard deviations of 3, 3, 2, 2 and 1, respectively. While the first two modes overlap significantly, the fifth mode stands isolated as shown in Fig. 4. We train different GAN models using $200,000$ samples from this distribution and generate $65,536$ data points from each model. In order to compare the learned distribution with the ground truth distributions, we first estimate them using bins over the data points and create the histograms. These histograms are carefully created using different bin sizes and the best bin (found to be $0.1$) is chosen. Then, we use Chi-square distance and the KL-divergence to compute distance between the two histograms. From Fig. 4 and Tab. 1 it is evident that MAD-GAN is able to capture all the clustered modes which includes significantly overlapped modes as well. MAD-GAN obtains the minimum value in terms of both Chi-square distance and the KL-divergence. In this experiment, both MAD-GAN and MA-GAN used four generators. In the case of InfoGAN, we used $5$ dimensional categorical variable, which provides the best result.

To understand the effect of varying the number of generators in MAD-GAN, we use the same synthetic experiment setup, i.e. the real data distribution is same GMM with $5$ Gaussians. For better non-parametric estimation, we use $1$ million sample points from real distribution (instead of $65,536$). We generate equal number of points from each of the generators such that they sum up to $1$ million. The results are shown in Fig. 5 and corresponding Tab. 2. It is quite clear that as the number of generators are increased up to $4$, the sampling keeps getting more realistic. In case when multiple modes are significantly overlapped/clustered, a generator can capture cluster of modes. Therefore, for this real data distribution, $4$ generators are enough to capture all the $5$ modes. With $5$ or more generators, all the modes were still captured, but the two overlapping modes have more than two generation peaks. This is mainly because multiple generators are capturing this region and all the generators (mixture components) were assigned equal weights during sampling.

Other works using more than one generators [21, 4] also use the number of generators as a hyper-parameter as knowing a-priori the number of modes in a real-world data (e.g. images) in itself is an open problem.

5.2 Stacked and Compositional MNIST

We now perform experiments on a more challenging setup, similar to [7, 22], in order to examine and compare MAD-GAN with other GAN variants. [22] created a Stacked-MNIST dataset with $25,600$ samples where each sample has three channels stacked together with a random MNIST digit in each of them. Thus, it creates $1000$ distinct modes in the data distribution. [22] used a stripped down version of the generator and discriminator pair to reduce the modeling capacity. We do the same for fair comparisons and used the same architecture as mentioned in their paper. Similarly, [7] created Compositional-MNIST whereby they took $3$ random MNIST digits and place them at the $3$ quadrants of a $64\times 64$ dimensional image. This also resulted in a data distribution with $1000$ hand-designed modes. The distribution of the resulting generated samples was estimated using a pretrained MNIST classifier to classify each of the digits either in the channels or the quadrants to decide the mode it belongs to.

Tables 3 and 4 provide comparison of our method with variants of GAN in terms of KL divergence and the number of modes recovered for the Stacked and Compositional MNIST datasets, respectively. In Stacked-MNIST, as evident from the Tab. 3, MAD-GAN outperforms all other variants of GAN in both the criteria. Interestingly, in the case of Compositional-MNIST, as shown in Tab. 4, MAD-GAN, WGAN and Unrolled GAN were able to recover all the $1000$ modes. However, in terms of KL divergence, the distribution generated by MAD-GAN is the closest to the true data distribution.

5.3 Diverse Samples for Image-to-Image Translation and Comparison to InfoGAN

Here we present experimental results on the challenging task of image-to-image translation [14] which uses conditional variant of GANs [23]. Conditional GAN for this task is known to learn the delta distribution, thus, generates the same image irrespective of the variations in the input noise vector. Generating diverse samples in this setting in itself is an open problem. We show that MAD-GAN is able to generate diverse samples in these experiments as well. We use three generators for MAD-GAN experiments and show three diverse generations. Note that, we do not claim to capture all the possible modes present in the data distribution because firstly we cannot estimate the number of modes a priori, and secondly, even if we could, we do not know how diverse the generations would be after using certain number of generators. We follow the same approach as [14] and employ patch based conditional GAN.

We compare MAD-GAN with InfoGAN [8] in these experiments as it is closest to our approach and can be used in image-to-image translation task. Theoretically, latent codes in InfoGAN should enable diverse generations. However, InfoGAN can only be used when the bias introduced by the categorical variables have significant impact on the generator network. For image-to-image translation and high resolution generations, the categorical variable does not have sufficient impact on the generations. As will be seen shortly, we validate this hypothesis by comparing our method with InfoGAN for this task. For the InfoGAN generator, to capture three kinds of distinct modes, the categorical code is chosen to take three values. Since we are dealing with images, in this case, the categorical code is a 2D matrix in which we set one third of the entries to 1 and remaining to 0 for each category. The generator is fed input image along with categorical code appended channel wise to the image. Architecture of the Q network is same as that of the pix2pix discriminator [14], except that the output is a vector of size $3$ for the prediction of the categorical codes. Note that, we tried different variations of the categorical codes but did not observe any significant variation in the generations.

Fig. 6 shows generations by MAD-GAN and InfoGAN for the edges-to-handbags task, where given the edges of handbags, the objective is to generate real looking handbags. Clearly, each MAD-GAN generator is able to produce meaningful images but different from remaining generators in terms of color, texture, and patterns. However, InfoGAN generations are almost the same for all the three categorical codes. The results shown for InfoGAN are obtained by not sharing the discriminator and Q network parameters.

To make our baseline as strong as possible, we did some more experiments with InfoGAN for the edges-to-handbags task. For Fig. 7, we did two experiments by sharing all the initial layers of the discriminator and Q network. In the first experiment, the input is the categorical code besides the conditional image. In the second experiment, noise is also added as an input. The architecture details are given in Appendix C.3.2. In Fig. 7, we show the results of both these experiments side by side. There are not much perceivable changes as we vary the categorical code values. Generator simply learn to ignore the input noise as was also pointed by [14].

In addition, in Fig. 8, we show diverse generations for the night-to-day task, where given night images of places, the objective is to generate their corresponding day images. As can be seen, the generated day images in Fig. 8 differ in terms of lighting conditions, sky patterns, weather conditions, and many other minute yet useful cues.

5.4 Diverse-Class Data Generation

To further explore the mode capturing capacity of MAD-GAN, we experimented with a much more challenging task of diverse-class data generation. In detail, we trained MAD-GAN (three generators) on a combined dataset consisting of various highly diverse images such as islets, icebergs, broadleaf-forest, bamboo-forest, and bedroom, obtained from the Places dataset [32]. Images were randomly selected from each of them, creating a training dataset of $24,000$ images. The generators have the same architecture as that of DCGAN. In this case, as the images in the dataset belong to different classes, we did not share the generator parameters. As shown in Fig. 1, to our surprise, we found that even in this highly challenging setting, the generations from different generators belong to different classes. This clearly indicates that the generators in MAD-GAN are able to disentangle inter-class variations. In addition, each generator for different noise input is able to generate diverse samples, indicating intra-class diversity.

5.5 Diverse Face Generation

Here we show diverse face generations (CelebA dataset) using MAD-GAN where we use DCGAN [26] as each of our three generators. Again, we use the same setting as provided in DCGAN. The high quality face generations are shown in the Fig. 10.

To get better understanding about the possible diversities, we show additional generations in Fig. 9.

5.6 Unsupervised Representation Learning

Similar to DCGAN [26], we train our framework using SVHN dataset [24]. The trained discriminator is used to extract features. Using these features, we train an SVM for the classification task. For the MAD-GAN, with three generators, we obtained misclassification error of $17.5\%$ which is almost $5\%$ better than the results reported by DCGAN ($22.48\%$). This clearly indicates that our framework is able to learn a better feature space in an unsupervised setting.

6 Conclusion

We presented a very simple and effective framework, Multi-Agent Diverse GAN (MAD-GAN), for generating diverse and meaningful samples. We showed the efficacy of our approach and compared it with various variants of GAN that it captures diverse modes while producing high quality samples. We presented a theoretical analysis of MAD-GAN with conditions for global optimality. Looking forward, an interesting future direction would be to estimate a priori the number of generators needed for a particular dataset. It is not clear how to do that given that we do not have access to the true data distribution. In addition, we would also like to theoretically understand the limiting cases that depend on the relationship between the number of generators and the complexity of the data distribution. Another interesting direction would be to exploit different generators such that their combinations can be used to capture diverse modes.

7 Acknowledgements

This work was supported by the EPSRC, ERC grant ERC-2012-AdG 321162-HELIOS, EPSRC grant Seebibyte EP/M013774/1 and EPSRC/MURI grant EP/N019474/1.

Appendix

Here, we first give better insights about the Theorem 1 and discuss how and when MAD-GAN leads to diverse generations. In Appendix B, we introduce another way of getting different generators to generate diverse samples. We introduce intuitive similarity based competing objective (MAD-GAN-Sim) which encourages different generators to generate diverse samples. Finally in Appendix C, we provide architecture details and data preparations for all the experiments reported for MAD-GAN and MAD-GAN-Sim.

Appendix A Insights for Diversity in MAD-GAN

One obvious question that could arise is that is it possible that all the generators learn to capture the same mode?. The short answer is, theoretically yes and in practice no. Let us begin with the discussion to understand this. Theoretically, according to Theorem 1 if $p_{g_{i}}=p_{d}$, for all $i$, then also the minimum objective value can be achieved. This implies, in worst case, MAD-GAN would perform same as the standard GAN. However, as discussed below, this is possible in following highly unlikely situations:

• •

all the generators always generate exactly similar samples so that the discriminator is not able to differentiate them. In this case, the discriminator will learn a uniform distribution over the generator indices, thus, the gradients passed through the discriminator will be exactly the same for all the generators. However, this situation in general is not possible as all the generators are initialized differently. Even a slight variation in the samples from the generators will be enough for the discriminator to identify them and pass different gradient information to each generator. In addition, the objective function of generators is only to generate real samples, thus, there is nothing that encourages them to generate exactly the same samples.

• •

the discriminator does not have enough capacity to learn the optimal parameters. This is in contrast to the assumption made in Theorem 1, which is that the discriminator is optimal. Thus, it should have enough capacity to learn a feature representation such that it can correctly identify samples from different generators. In practice, this is a very easy task and we did not have to modify anything up to the feature representation stage of the architecture of the discriminator. We used the standard architectures (explained in Appendix C) for all the tasks.

Hence, with random initializations and sufficient capacity generator/discriminator, we can easily avoid the trivial solution in which all the generators focus on exactly the same region of the true data distribution. This has been very clearly supported by various experiments showing diverse generations by MAD-GAN.

Appendix B Similarity based competing objective

We have discussed the MAD-GAN architecture using generator identification based objective. In this section, we propose a different extension to the standard GAN : similarity based competing objective, which we call as MAD-GAN-Sim. Here, we augment the GAN objective function with a diversity enforcing term. It ensures that the generations from different generators are diverse where the diversity depends on a user-defined task-specific function.

The architecture is same as MAD-GAN discussed in Section 4.1 (refer Fig. 11).

B.1 Approach

The approach presented here is motivated by the fact that the samples from different modes must look different. For example, in the case of images, these samples should differ in terms of texture, color, shading, and various other cues. Thus, different generators must generate dissimilar samples where the dissimilarity comes from a task-specific function. Before delving into the details, let us first define some notations in order to avoid clutter. We denote $\theta_{g}^{i}$ as the parameters of the $i$-th generator. The set of generators is denoted as $K=\{1,\cdots,k\}$. Given random noise $z$ to the $i$-th generator, the corresponding generated sample $G_{i}(z;\theta_{g}^{i})$ is denoted as $g_{i}(z)$. Using these notations and following the above discussed intuitions, we impose following constraints over the $i$-th generator while updating its parameters:

[TABLE]

where, $\phi(g_{i}(z))$ denotes the mapping of the generated image $g_{i}(z)$ by the $i$-th generator into a feature space and $\Delta(.,.)\in[0,1]$ is the similarity function. Higher the value of $\Delta(.,.)$ more similar the arguments are. Intuitively, the above set of constraints ensures that the discriminator score for each generator should be higher than all other generators with a margin proportional to the similarity score. If the samples are similar, the margin increases and the constraints become more active. We use unsupervised learning based representation as our mapping function $\phi(.)$. Precisely, given a generated sample $g_{i}(z)$, $\phi(g_{i}(z))$ is the feature vector obtained using the discriminator of our framework. This is motivated by the feature matching based approach to improve the stability of the training of GANs [28]. The $\Delta(.,.)$ function used in this work is the standard cosine similarity based function. The above mentioned constraints can be satisfied by maximizing an equivalent unconstrained objective function as defined below:

[TABLE]

where, $f(a)=\min(0,a)$, $\psi_{i}=\phi(g_{i}(z))$, and $\psi_{j}=\phi(g_{j}(z))$. Intuitively, if the argument of $f(.)$ is positive, then the desirable constraint is satisfied and there is no need to do anything. Otherwise, maximize the argument with respect to $\theta_{g}^{i}$. Note that instead of using all the constraints independently, we use the average of all of them. Another approach would be to use the constraint corresponding to the $j$-th generator that maximally violates the set of constraints shown in Eq. B.1. Experimentally we found that the training process of the average constraint based objective is more stable than the maximum violated constraint based objective. The intuition behind using these constraints comes from the well know $1$-slack formulation of the structured SVM framework [15, 29]. Thus, the overall objective for the $i$-th generator is:

[TABLE]

where $\lambda\geq 0$ is the hyperparameter. Algorithm 1 shows how to compute gradients corresponding to different generators for the above mentioned objective function. Notice that, once sampled, the same $z$ is passed through all the generators in order to enforce constraints over a particular generator (as shown in Eq. B.1). However, in order for constraints to not to contradict with each other while updating another generator, a different $z$ is sampled again from the $p_{z}$. The Algorithm 1 is shown for the batch of size one which can be trivially generalized for any given batch sizes. In the case of discriminator, the gradients will have exactly the same form as the standard GAN objective. The only difference is that in this case the fake samples are being generated by $k$ generators, instead of one.

B.2 Experiments

We present the efficacy of MAD-GAN-Sim on the real world datasets.

B.3 Diverse Samples for Image-to-Image Translation

We show diverse and highly appealing results using the diversity promoting objective. We use cosine based similarity to enforce diverse generations, an important criteria for real images. As before, we show results for the following two situations where diverse solution is useful: (1) given the edges of handbags, generate real looking handbags as in Fig. 12; and (2) given night images of places, generate their equivalent day images as in Fig. 13. We clearly notice that each generator is able to produce meaningful and diverse images.

B.4 Unsupervised Representation Learning

We do the same experiment using SVHN as done in Section 5.6. For MAD-GAN-Sim we obtained the misclassification error of 18.3% which is better than DCGAN (22.48%). It clearly indicates that MAD-GAN-Sim is able to learn better feature representation in an unsupervised setting.

Appendix C Network Architectures and Parameters

Here we provide all the details about the architectures and the parameters used in various experiments. For the experiment concerning non-parametric density estimation, the MAD-GAN parameters are randomly initialized using xavier initialization with normal distributed random sampling [12]. For all the other experiments, the initialization done is same as the base architecture used to adapt MAD-GAN.

C.1 Non-Parametric Density Estimation

Architecture Details:

The generator has two fully connected hidden layers with $128$ neurons (each of which are followed by exponential linear unit) and fully connected outer layer. In case of MAD-GAN and MA-GAN, we used $4$ generators with parameters of first two layers shared. Generator generates 1D samples. Input to each generator is a uniform noise $U(-1,1)$ of $64$ dimension. In case of InfoGAN, $5$ dimensional categorical code is further concatenated with the uniform noise to form the input. The categorical code is randomly sampled from the multinomial distribution. The discriminator architecture for respective networks is shown in Tab. 5. Mode-Regularized GAN architecture has encoder, BEGAN has encoder and decoder, and InfoGAN has Q Network whose details are also present in Tab. 5.

MAD-GAN has multi-label cross entropy loss. MA-GAN has binary cross entropy loss. For training, we use Adam optimizer with batch size of $128$ and learning rate of $1e-4$. In each mini batch, for MAD-GAN we have $128$ samples from each of the generators as well as real distribution, while for MA-GAN $128$ samples are chosen from real distribution as well as all the generators combined.

Dataset Generation

We generated synthetic 1D data using GMM with $5$ Gaussians and select their means at $10,20,60,80$ and $110$. The standard deviation used is $3,3,2,2$ and $1$. The first two modes overlap significantly while the fifth one is peaky and stands isolated.

C.2 Stacked and compositional MNIST Experiments

Architecture details:

The architecture for stacked-MNIST is similar to the one used in [22]. Please refer to the Tab. 6 for generator architecture and Tab. 7 for discriminator architecture and Q network architecture of InfoGAN. The architecture for compositional-MNIST experiment is same as DCGAN [26]. Please refer to the Tab. 8 for discriminator architecture and Q network architecture of InfoGAN. In both the experiments, Q network of InfoGAN shares all except the last layer with the discriminator.

Dataset preparation:

MNIST database of hand written digits are used for both the tasks.

C.3 Image-to-Image Translation

C.3.1 MAD-GAN/MAD-GAN-Sim

Architecture details:

The network architecture is adapted from [14] and the experiments were conducted with the U-Net architecture and patch based discriminator.

In more detail, let Ck denote a Convolution-BatchNorm-ReLU layer with k filters and CDk represent a Convolution-BatchNorm-Dropout-ReLU layer with a dropout rate of $50$%. All Convolutions are $4\times 4$ spatial filters with a stride of $2$. Convolutions in the encoder, and in the discriminator, downsample by a factor of $2$, whereas in the decoder they upsample by a factor of $2$.

Generator Architectures

We used the U-Net generator based architecture from [14] as follows:

• •

U-Net Encoder: C64-C128-C256-C512-C512-C512-C512-C512

• •

U-Net Decoder: CD512-CD1024-CD1024-C1024-C1024-C512-C256-C128. Note that, in case of MAD-GAN, the last layer does not share parameters with other generators.

After the last layer in the decoder, a convolution is applied to map to the number of output channels to $3$, followed by a tanh function. BatchNorm is not applied to the first C64 layer in the encoder. All ReLUs in the encoder are leaky, with a slope of $0.2$, while ReLUs in the decoder are not leaky. The U-Net architecture has skip-connections between each layer $i$ in the encoder and layer $n-i$ in the decoder, where $n$ is the total number of layers. The skip connections concatenate activations from layer $i$ to layer $n-i$. This changes the number of channels in the decoder.

Discriminator Architectures

The patch based $70\times 70$ discriminator architecture was used in this case : C64-C128-C256-C512.

Diversity term

• •

MAD-GAN: After the last layer, a convolution is applied to map the output layer to the dimension of $k+1$ (where $k$ is the number of generators in MAD-GAN) followed by the softmax layer for the normalization.

• •

MAD-GAN-Sim: After the last layer, a convolution is applied to map to a 1 dimensional output followed by a Sigmoid function. For the unsupervised feature representation $\phi(.)$, the feature activations from the penultimate layer C256 of the discriminator was used as the feature activations for the computation of the cosine similarity.

For the training, we used Adam optimizer with learning rate of $2e-4$ (for both generators and discriminator), $\lambda_{L1}=10$ (hyperparameter corresponding to the $L_{1}$ regularizer), $\lambda=1e-3$ (corresponding to MAD-GAN-Sim), and batch size of $1$.

C.3.2 InfoGAN

The network architecture is adapted from [14] and the experiments were conducted with the U-Net architecture and patch based discriminator.

Generator Architectures

The U-Net generator is exactly same as in [14] except that the number of input channels are increased from $3$ to $4$. For the experiment done for Fig. 7, to take noise as input, input channels are increased to $5$ (one extra input channel for noise).

Discriminator Architectures

The discriminator is exactly same as in [14]: C64-C128-C256-C512

Q network Architectures

The Q network architecture is C64-C128-C256-C512-Convolution3-Convolution3. Here first Convolution3 gives a output of $30\times 30$ patches with $3$ channels while second Convolution3 just gives $3$ dimensional output. All the layers except last two are shared with the discriminator to perform the experiments for Fig. 7.

Diversity term

To capture three kinds of distinct modes, the categorical code can take three values. Hence, in this case, the categorical code is a 2D matrix in which one third of entries are set to $1$ and remaining to [math] for each category. The generator is fed input image along with categorical code appended channel wise to the image. For the experiment done for Fig. 7, to take noise as input, the generator input is further channel wise appended with a 2D matrix of normal noise.

For the training, we used Adam optimizer with learning rate of $2e-4$ (for both generator and discriminator), $\lambda_{L1}=10$ (hyperparameter corresponding to the $L_{1}$ regularizer) and batch size of $1$.

Dataset Preparation:

• •

Edges-to-Handbags: We used $137,000$ Amazon handbag images from [33]. The random split into train and test was kept the same as done by [33].

• •

Night-to-Day: We used $17,823$ training images extracted from 91 webcams. We thank Jun-Yan Zhu for providing the dataset.

C.4 Diverse-Class Data Generation

Architecture details:

The network architecture is adapted from DCGAN [26]. Concretely, the discriminator architecture is described in Tab. 11 and the generator architecture in Tab. 10. We use three generators without sharing any parameter. The residual layers helped in improving the image quality since the data manifold was much more complicated and the discriminator needed more capacity to accommodate it.

Diversity terms

For the training, we used Adam optimizer with the learning rate of $2e-4$ (both generator and discriminator) and batch size of $64$.

Dataset preparation:

Training data is obtained by combining dataset consisting of various highly diverse images such as islets, icebergs, broadleaf-forest, bamboo-forest and bedroom, obtained from the Places dataset [32]. To create the training data, images were randomly selected from each of them, creating a dataset consisting of $24,000$ images.

C.5 Diverse Face Generations with DCGAN

Architecture details:

The network architecture is adapted from DCGAN [26]. Concretely, the discriminator architecture is described in Tab. 11 and the generator architecture in Tab. 10. In this case all the parameters of the generators except the last layer were shared. The residual layers helped in improving the image quality since the data manifold and the manifolds of each of the generators was much more complicated and the discriminator needed more capacity to accommodate it.

Diversity terms

For the training, we used Adam optimizer with the learning rate of $2e-4$ (both generator and discriminator) and batch size of $64$.

Dataset preparation:

We used CelebA dataset as mentioned for face generation based experiments. For Image generation all the images ($14,197,122$) from the Imagenet-1k dataset [9] were used to train the DCGAN with $3$ Generators alongside the MAD-GAN objective. The images from both CelebA and Imagenet-1k were resized into $64\times 64$.

C.6 Unsupervised Representation Learning

Architecture details:

Our architecture uses the one proposed in DCGAN [26]. Similar to the DCGAN experiment on SVHN dataset ($32\times 32\times 3$) [24], we removed the penultimate layer of generator (second last row in Tab. 10) and first layer of discriminator (first convolution layer in Tab. 9).

Classification task:

We trained our model on the available SVHN dataset [24]. For feature extraction using discriminator, we followed the same method as mentioned in the DCGAN paper [26]. The features were then used for training a regularized linear L2-SVM. The ablation study is presented in Tab. 13.

Dataset preparation:

We used SVHN dataset [24] consisting of $73,257$ digits for the training, $26,032$ digits for the testing, and $53,1131$ extra training samples. As done in DCGAN [26], we used $1000$ uniformly class distributed random samples for training, $10,000$ samples from the non-extra set for validation and $1000$ samples for testing.

For the training, we used Adam optimizer with learning rate of $2e-4$ (both generator and discriminator), $\lambda=1e-4$ (competing objective), and batch size of $64$.