Limited-Memory LRSGA: An Iterative Method for Computing Nash Equilibria in Competitive Optimization Problems

TL;DR

LMLRSGA is a limited-memory iterative method designed to efficiently approximate Nash equilibria in high-dimensional differentiable games, such as GAN training, by combining first-order updates with symplectic second-order corrections.

Contribution

It introduces a limited-memory variant of LRSGA that is suitable for high-dimensional models and provides spectral stability analysis and empirical evaluation on GANs.

Findings

LMLRSGA achieves stable training dynamics in GANs.

Spectral diagnostics reveal insights into training stability.

Empirical results on MNIST and FashionMNIST demonstrate effectiveness.

Abstract

We introduce LMLRSGA, a limited memory variant of Low Rank Symplectic Gradient Adjustment (LRSGA) for differentiable games. It is an iterative scheme for approximating Nash equilibria with first order like cost while retaining the stabilizing effect of symplectic second order corrections via low rank information. By storing only a limited history of curvature pairs, LMLRSGA is well suited to high parameter competitive models such as GANs. In particular, we provide a per iteration spectral stability condition for LRSGA near Nash equilibria, a limited memory implementation (LMLRSGA) based on adapted two loop recursions together with a local convergence analysis for fixed history length, and an empirical evaluation on GAN training on MNIST and FashionMNIST, including spectral diagnostics of the training dynamics.