Riemannian SVRG: Fast Stochastic Optimization on Riemannian Manifolds

TL;DR

This paper introduces RSVRG, a novel variance reduction algorithm for stochastic optimization on Riemannian manifolds, providing the first non-asymptotic complexity analysis for nonconvex Riemannian problems.

Contribution

The paper presents RSVRG, the first provably fast stochastic Riemannian optimization method with a non-asymptotic analysis for nonconvex functions, extending variance reduction techniques to manifold settings.

Findings

RSVRG inherits SVRG advantages with curvature-dependent convergence factors.

First non-asymptotic complexity analysis for nonconvex Riemannian optimization.

Application to variance reduced PCA with transparent convergence analysis.

Abstract

We study optimization of finite sums of geodesically smooth functions on Riemannian manifolds. Although variance reduction techniques for optimizing finite-sums have witnessed tremendous attention in the recent years, existing work is limited to vector space problems. We introduce Riemannian SVRG (RSVRG), a new variance reduced Riemannian optimization method. We analyze RSVRG for both geodesically convex and nonconvex (smooth) functions. Our analysis reveals that RSVRG inherits advantages of the usual SVRG method, but with factors depending on curvature of the manifold that influence its convergence. To our knowledge, RSVRG is the first provably fast stochastic Riemannian method. Moreover, our paper presents the first non-asymptotic complexity analysis (novel even for the batch setting) for nonconvex Riemannian optimization. Our results have several implications; for instance, they offer a Riemannian perspective on variance reduced PCA, which promises a short, transparent convergence analysis.