Riemannian Hamiltonian methods for min-max optimization on manifolds

TL;DR

This paper introduces Riemannian Hamiltonian methods for min-max optimization on manifolds, providing convergence analysis and applications to neural network training, Wasserstein distances, and GANs.

Contribution

It proposes a novel Riemannian Hamiltonian framework for min-max problems, with convergence guarantees and extensions to stochastic and regularized settings.

Findings

Effective in training neural networks robustly

Applicable to Wasserstein distance computations

Demonstrates convergence in various applications

Abstract

In this paper, we study min-max optimization problems on Riemannian manifolds. We introduce a Riemannian Hamiltonian function, minimization of which serves as a proxy for solving the original min-max problems. Under the Riemannian Polyak--{\L}ojasiewicz condition on the Hamiltonian function, its minimizer corresponds to the desired min-max saddle point. We also provide cases where this condition is satisfied. For geodesic-bilinear optimization in particular, solving the proxy problem leads to the correct search direction towards global optimality, which becomes challenging with the min-max formulation. To minimize the Hamiltonian function, we propose Riemannian Hamiltonian methods (RHM) and present their convergence analyses. We extend RHM to include consensus regularization and to the stochastic setting. We illustrate the efficacy of the proposed RHM in applications such as subspace robust Wasserstein distance, robust training of neural networks, and generative adversarial networks.