Online Kernel based Generative Adversarial Networks

TL;DR

This paper introduces OKGAN, a kernel-based non-parametric discriminator for GANs, which improves training stability, reduces mode collapse, and offers better theoretical guarantees, achieving superior or comparable results on standard datasets.

Contribution

The paper proposes a novel online kernel-based discriminator for GANs, addressing training challenges and improving stability and performance.

Findings

OKGAN mitigates mode collapse and cycling.

OKGAN shows better reverse KL-divergence on synthetic data.

OKGAN achieves comparable performance on MNIST, SVHN, CelebA.

Abstract

One of the major breakthroughs in deep learning over the past five years has been the Generative Adversarial Network (GAN), a neural network-based generative model which aims to mimic some underlying distribution given a dataset of samples. In contrast to many supervised problems, where one tries to minimize a simple objective function of the parameters, GAN training is formulated as a min-max problem over a pair of network parameters. While empirically GANs have shown impressive success in several domains, researchers have been puzzled by unusual training behavior, including cycling so-called mode collapse. In this paper, we begin by providing a quantitative method to explore some of the challenges in GAN training, and we show empirically how this relates fundamentally to the parametric nature of the discriminator network. We propose a novel approach that resolves many of these issues by relying on a kernel-based non-parametric discriminator that is highly amenable to online training---we call this the Online Kernel-based Generative Adversarial Networks (OKGAN). We show empirically that OKGANs mitigate a number of training issues, including mode collapse and cycling, and are much more amenable to theoretical guarantees. OKGANs empirically perform dramatically better, with respect to reverse KL-divergence, than other GAN formulations on synthetic data; on classical vision datasets such as MNIST, SVHN, and CelebA, show comparable performance.