Approximate Last Iterate Convergence in Overparameterized GANs

TL;DR

This paper proves that certain optimization methods for overparameterized GANs converge to a neighborhood of the optimum at the last iterate, with the neighborhood size decreasing as network width increases, improving upon previous average convergence guarantees.

Contribution

It establishes last iterate convergence for Implicit Update and Predictive Methods in overparameterized GANs, showing convergence to a shrinking neighborhood.

Findings

Last iterate convergence to a neighborhood around the optimum.

Neighborhood size decreases with network width.

Contrasts with prior average convergence results.

Abstract

In this work, we showed that the Implicit Update and Predictive Methods dynamics introduced in prior work satisfy last iterate convergence to a neighborhood around the optimum in overparameterized GANs, where the size of the neighborhood shrinks with the width of the neural network. This is in contrast to prior results, which only guaranteed average iterate convergence.