Demystifying MMD GANs

TL;DR

This paper clarifies bias issues in MMD GANs and Wasserstein GANs, analyzes kernel choices, and introduces a new convergence measure, demonstrating simpler, faster training with comparable performance.

Contribution

It provides a theoretical clarification on bias in GAN loss functions, analyzes kernel choices for MMD critics, and proposes a new convergence metric and training strategies.

Findings

MMD GANs can use smaller critics than Wasserstein GANs.

Gradient estimators are unbiased, but generator gradients are biased when learning from samples.

Kernel Inception Distance effectively measures GAN convergence.

Abstract

We investigate the training and performance of generative adversarial networks using the Maximum Mean Discrepancy (MMD) as critic, termed MMD GANs. As our main theoretical contribution, we clarify the situation with bias in GAN loss functions raised by recent work: we show that gradient estimators used in the optimization process for both MMD GANs and Wasserstein GANs are unbiased, but learning a discriminator based on samples leads to biased gradients for the generator parameters. We also discuss the issue of kernel choice for the MMD critic, and characterize the kernel corresponding to the energy distance used for the Cramer GAN critic. Being an integral probability metric, the MMD benefits from training strategies recently developed for Wasserstein GANs. In experiments, the MMD GAN is able to employ a smaller critic network than the Wasserstein GAN, resulting in a simpler and faster-training algorithm with matching performance. We also propose an improved measure of GAN convergence, the Kernel Inception Distance, and show how to use it to dynamically adapt learning rates during GAN training.