ODE Analysis of Stochastic Gradient Methods with Optimism and Anchoring for Minimax Problems

TL;DR

This paper provides a theoretical analysis of stochastic gradient methods with optimism and anchoring for minimax problems, particularly in the context of GAN training dynamics, using differential equations for convergence insights.

Contribution

It introduces and analyzes optimistic and anchored stochastic gradient methods for convex-concave minimax problems, extending understanding of their convergence properties.

Findings

simGD converges with stochastic sub-gradients under strict convexity

Optimistic simGD converges with full gradients when using a separate optimism rate

Anchored simGD achieves convergence with stochastic subgradients

Abstract

Despite remarkable empirical success, the training dynamics of generative adversarial networks (GAN), which involves solving a minimax game using stochastic gradients, is still poorly understood. In this work, we analyze last-iterate convergence of simultaneous gradient descent (simGD) and its variants under the assumption of convex-concavity, guided by a continuous-time analysis with differential equations. First, we show that simGD, as is, converges with stochastic sub-gradients under strict convexity in the primal variable. Second, we generalize optimistic simGD to accommodate an optimism rate separate from the learning rate and show its convergence with full gradients. Finally, we present anchored simGD, a new method, and show convergence with stochastic subgradients.