Mirror Descent on Riemannian Manifolds

TL;DR

This paper extends the Mirror Descent optimization method to Riemannian manifolds, introducing a Riemannian Mirror Descent framework and its stochastic variant with convergence guarantees, applicable to large-scale manifold optimization.

Contribution

It generalizes Mirror Descent to Riemannian manifolds, develops a stochastic version, and provides convergence analysis, broadening the scope of first-order optimization methods.

Findings

RMD reduces to Curvilinear Gradient Descent on the Stiefel manifold.

Stochastic RMD effectively handles large-scale manifold optimization.

Non-asymptotic convergence guarantees are established for both RMD and stochastic RMD.

Abstract

Mirror Descent (MD) is a scalable first-order method widely used in large-scale optimization, with applications in image processing, policy optimization, and neural network training. This paper generalizes MD to optimization on Riemannian manifolds. In particular, we develop a Riemannian Mirror Descent (RMD) framework via reparameterization and further propose a stochastic variant of RMD. We also establish non-asymptotic convergence guarantees for both RMD and stochastic RMD. As an application to the Stiefel manifold, our RMD framework reduces to the Curvilinear Gradient Descent (CGD) method proposed in [26]. Moreover, when specializing the stochastic RMD framework to the Stiefel setting, we obtain a stochastic extension of CGD, which effectively addresses large-scale manifold optimization problems.