Empirical Analysis of Overfitting and Mode Drop in GAN Training

TL;DR

This paper empirically investigates overfitting and mode drop in GAN training, revealing that removing stochasticity leads to overfitting and minimal mode drop, challenging previous beliefs about GAN memorization and mode collapse.

Contribution

It demonstrates that GANs can overfit and memorize training data when stochasticity is removed, providing new insights into GAN training dynamics.

Findings

Removing stochasticity causes GAN overfitting.

GANs can memorize training data.

Mode drop is influenced by training procedures.

Abstract

We examine two key questions in GAN training, namely overfitting and mode drop, from an empirical perspective. We show that when stochasticity is removed from the training procedure, GANs can overfit and exhibit almost no mode drop. Our results shed light on important characteristics of the GAN training procedure. They also provide evidence against prevailing intuitions that GANs do not memorize the training set, and that mode dropping is mainly due to properties of the GAN objective rather than how it is optimized during training.