Adaptive Single-Loop Methods for Stochastic Minimax Optimization on Riemannian Manifolds

TL;DR

This paper introduces adaptive single-loop algorithms for stochastic minimax optimization on Riemannian manifolds, eliminating the need for prior knowledge of problem parameters and achieving improved convergence rates.

Contribution

The paper develops the first single-loop adaptive methods for Riemannian minimax problems, with convergence guarantees and practical effectiveness demonstrated through experiments.

Findings

Deterministic method attains $ ext{O}(rac{1}{ extepsilon^2})$ convergence.

Stochastic method achieves $ ext{O}(rac{1}{ extepsilon^6})$ complexity, improved to $ ext{O}(rac{1}{ extepsilon^4})$ under smoothness.

Numerical experiments show the methods are effective in real-world applications.

Abstract

Stochastic minimax optimization on Riemannian manifolds has recently attracted significant attention due to its broad range of applications, such as robust training of neural networks and robust maximum likelihood estimation. Existing optimization methods for these problems typically require selecting stepsizes based on prior knowledge of specific problem parameters, such as Lipschitz-type constants and (geodesic) strong concavity constants. Unfortunately, these parameters are often unknown in practice. To overcome this issue, we develop single-loop adaptive methods that automatically adjust stepsizes using cumulative Riemannian (stochastic) gradient norms. We first propose a deterministic single-loop Riemannian adaptive gradient descent ascent method and show that it attains an $\epsilon$-stationary point within $O(\epsilon^{-2})$ iterations. This deterministic method is of independent interest and lays the foundation for our subsequent stochastic method. In particular, we propose the Riemannian stochastic adaptive gradient descent ascent method, which finds an $\epsilon$-stationary point in $O(\epsilon^{-6})$ iterations. Under additional second-order smoothness, this iteration complexity is further improved to $O(\epsilon^{-4})$, which even outperforms the corresponding complexity result in Euclidean space. Some numerical experiments on real-world applications are conducted, including the regularized robust maximum likelihood estimation problem, and the robust training of neural networks with orthonormal weights. The results are encouraging and demonstrate the effectiveness of adaptivity in practice.