Zeroth-order Riemannian Averaging Stochastic Approximation Algorithms

TL;DR

This paper introduces Zo-RASA, a new zeroth-order stochastic optimization algorithm on Riemannian manifolds that achieves optimal sample complexity with practical improvements like retractions and vector transport.

Contribution

It proposes Zo-RASA, combining Riemannian averaging and novel analysis techniques, to improve efficiency and practicality in stochastic Riemannian optimization.

Findings

Achieves optimal sample complexity for epsilon-approximate solutions.

Uses retractions and vector transport to reduce per-iteration complexity.

Provides new error bounds under geometric conditions.

Abstract

We present Zeroth-order Riemannian Averaging Stochastic Approximation (\texttt{Zo-RASA}) algorithms for stochastic optimization on Riemannian manifolds. We show that \texttt{Zo-RASA} achieves optimal sample complexities for generating $\epsilon$-approximation first-order stationary solutions using only one-sample or constant-order batches in each iteration. Our approach employs Riemannian moving-average stochastic gradient estimators, and a novel Riemannian-Lyapunov analysis technique for convergence analysis. We improve the algorithm's practicality by using retractions and vector transport, instead of exponential mappings and parallel transports, thereby reducing per-iteration complexity. Additionally, we introduce a novel geometric condition, satisfied by manifolds with bounded second fundamental form, which enables new error bounds for approximating parallel transport with vector transport.