On the Limitations of First-Order Approximation in GAN Dynamics

TL;DR

This paper analyzes GAN training dynamics in a simplified model, revealing that first-order discriminator updates can cause instability and mode collapse, highlighting limitations of common approximation methods.

Contribution

It provides a rigorous theoretical analysis showing the divergence caused by first-order discriminator steps in GANs, despite the model's non-convex nature.

Findings

Optimal discriminator leads to convergence

First-order approximation causes instability

First-order steps contribute to mode collapse

Abstract

While Generative Adversarial Networks (GANs) have demonstrated promising performance on multiple vision tasks, their learning dynamics are not yet well understood, both in theory and in practice. To address this issue, we study GAN dynamics in a simple yet rich parametric model that exhibits several of the common problematic convergence behaviors such as vanishing gradients, mode collapse, and diverging or oscillatory behavior. In spite of the non-convex nature of our model, we are able to perform a rigorous theoretical analysis of its convergence behavior. Our analysis reveals an interesting dichotomy: a GAN with an optimal discriminator provably converges, while first order approximations of the discriminator steps lead to unstable GAN dynamics and mode collapse. Our result suggests that using first order discriminator steps (the de-facto standard in most existing GAN setups) might be one of the factors that makes GAN training challenging in practice.