Your GAN is Secretly an Energy-based Model and You Should use Discriminator Driven Latent Sampling

TL;DR

This paper reveals that GANs can be viewed as energy-based models through their discriminator, and introduces Discriminator Driven Latent Sampling (DDLS), a method that improves sample quality efficiently by sampling in latent space.

Contribution

The paper demonstrates that the GAN discriminator defines an energy function and proposes DDLS, a practical latent space sampling method that enhances GAN sample quality without extra training.

Findings

DDLS significantly improves CIFAR-10 Inception Score from 8.22 to 9.09.

DDLS achieves state-of-the-art results in unconditional image synthesis.

Sampling in latent space is more efficient than pixel space methods.

Abstract

We show that the sum of the implicit generator log-density $\log p_g$ of a GAN with the logit score of the discriminator defines an energy function which yields the true data density when the generator is imperfect but the discriminator is optimal, thus making it possible to improve on the typical generator (with implicit density $p_g$). To make that practical, we show that sampling from this modified density can be achieved by sampling in latent space according to an energy-based model induced by the sum of the latent prior log-density and the discriminator output score. This can be achieved by running a Langevin MCMC in latent space and then applying the generator function, which we call Discriminator Driven Latent Sampling~(DDLS). We show that DDLS is highly efficient compared to previous methods which work in the high-dimensional pixel space and can be applied to improve on previously trained GANs of many types. We evaluate DDLS on both synthetic and real-world datasets qualitatively and quantitatively. On CIFAR-10, DDLS substantially improves the Inception Score of an off-the-shelf pre-trained SN-GAN~\citep{sngan} from $8.22$ to $9.09$ which is even comparable to the class-conditional BigGAN~\citep{biggan} model. This achieves a new state-of-the-art in unconditional image synthesis setting without introducing extra parameters or additional training.