Local Convergence of Gradient Descent-Ascent for Training Generative Adversarial Networks

TL;DR

This paper analyzes the local convergence behavior of gradient descent-ascent in training GANs with kernel-based discriminators, revealing how learning rates, regularization, and kernel bandwidth influence stability and convergence phases.

Contribution

It provides a linearized dynamical systems analysis of GDA for GANs with kernel discriminators, highlighting phase transitions and the impact of hyperparameters on convergence.

Findings

Identifies conditions for convergence, oscillation, or divergence of GDA.

Shows how learning rates and kernel bandwidth affect local stability.

Verifies theoretical predictions with numerical simulations.

Abstract

Generative Adversarial Networks (GANs) are a popular formulation to train generative models for complex high dimensional data. The standard method for training GANs involves a gradient descent-ascent (GDA) procedure on a minimax optimization problem. This procedure is hard to analyze in general due to the nonlinear nature of the dynamics. We study the local dynamics of GDA for training a GAN with a kernel-based discriminator. This convergence analysis is based on a linearization of a non-linear dynamical system that describes the GDA iterations, under an \textit{isolated points model} assumption from [Becker et al. 2022]. Our analysis brings out the effect of the learning rates, regularization, and the bandwidth of the kernel discriminator, on the local convergence rate of GDA. Importantly, we show phase transitions that indicate when the system converges, oscillates, or diverges. We also provide numerical simulations that verify our claims.