The Unusual Effectiveness of Averaging in GAN Training

TL;DR

This paper investigates the effectiveness of parameter averaging techniques, specifically Moving Average (MA) and Exponential Moving Average (EMA), in GAN training, providing theoretical insights and demonstrating improved stability and performance across various datasets.

Contribution

The paper offers the first theoretical analysis supporting EMA in GANs and empirically shows both MA and EMA improve GAN training stability and results.

Findings

EMA converges to limit cycles with vanishing amplitude in bilinear games.

Both MA and EMA improve GAN stability and scores across multiple datasets.

State-of-the-art results achieved on CIFAR-10 and high-quality CelebA images.

Abstract

We examine two different techniques for parameter averaging in GAN training. Moving Average (MA) computes the time-average of parameters, whereas Exponential Moving Average (EMA) computes an exponentially discounted sum. Whilst MA is known to lead to convergence in bilinear settings, we provide the -- to our knowledge -- first theoretical arguments in support of EMA. We show that EMA converges to limit cycles around the equilibrium with vanishing amplitude as the discount parameter approaches one for simple bilinear games and also enhances the stability of general GAN training. We establish experimentally that both techniques are strikingly effective in the non-convex-concave GAN setting as well. Both improve inception and FID scores on different architectures and for different GAN objectives. We provide comprehensive experimental results across a range of datasets -- mixture of Gaussians, CIFAR-10, STL-10, CelebA and ImageNet -- to demonstrate its effectiveness. We achieve state-of-the-art results on CIFAR-10 and produce clean CelebA face images.\footnote{~The code is available at \url{https://github.com/yasinyazici/EMA_GAN}}