Decentralized Riemannian Algorithm for Nonconvex Minimax Problems

TL;DR

This paper introduces the first decentralized algorithms for nonconvex minimax problems on Riemannian manifolds, specifically the Stiefel manifold, with theoretical convergence guarantees and empirical validation in neural network training.

Contribution

It develops deterministic and stochastic decentralized minimax algorithms for nonconvex Riemannian problems, providing the first convergence guarantees in this setting.

Findings

Deterministic method achieves $O( \\epsilon^{-2})$ gradient complexity.

Stochastic method achieves $O( \\epsilon^{-4})$ gradient complexity.

Algorithms demonstrate efficiency in neural network training over the Stiefel manifold.

Abstract

The minimax optimization over Riemannian manifolds (possibly nonconvex constraints) has been actively applied to solve many problems, such as robust dimensionality reduction and deep neural networks with orthogonal weights (Stiefel manifold). Although many optimization algorithms for minimax problems have been developed in the Euclidean setting, it is difficult to convert them into Riemannian cases, and algorithms for nonconvex minimax problems with nonconvex constraints are even rare. On the other hand, to address the big data challenges, decentralized (serverless) training techniques have recently been emerging since they can reduce communications overhead and avoid the bottleneck problem on the server node. Nonetheless, the algorithm for decentralized Riemannian minimax problems has not been studied. In this paper, we study the distributed nonconvex-strongly-concave minimax optimization problem over the Stiefel manifold and propose both deterministic and stochastic minimax methods. The Steifel manifold is a non-convex set. The global function is represented as the finite sum of local functions. For the deterministic setting, we propose DRGDA and prove that our deterministic method achieves a gradient complexity of $O( \epsilon^{-2})$ under mild conditions. For the stochastic setting, we propose DRSGDA and prove that our stochastic method achieves a gradient complexity of $O(\epsilon^{-4})$. The DRGDA and DRSGDA are the first algorithms for distributed minimax optimization with nonconvex constraints with exact convergence. Extensive experimental results on the Deep Neural Networks (DNNs) training over the Stiefel manifold demonstrate the efficiency of our algorithms.