Variance reduction for Riemannian non-convex optimization with batch size adaptation

TL;DR

This paper introduces an adaptive batch size variance reduction method for Riemannian non-convex optimization, improving convergence rates and complexity bounds over existing methods, with empirical validation on various tasks.

Contribution

It proposes a novel adaptive batch size strategy for variance reduction in Riemannian non-convex optimization, enhancing theoretical complexity bounds and simplifying convergence analysis.

Findings

Achieves lower total complexities for non-convex Riemannian optimization.

Provides simpler convergence analysis for R-SVRG.

Demonstrates effectiveness of adaptive batch size in experiments.

Abstract

Variance reduction techniques are popular in accelerating gradient descent and stochastic gradient descent for optimization problems defined on both Euclidean space and Riemannian manifold. In this paper, we further improve on existing variance reduction methods for non-convex Riemannian optimization, including R-SVRG and R-SRG/R-SPIDER with batch size adaptation. We show that this strategy can achieve lower total complexities for optimizing both general non-convex and gradient dominated functions under both finite-sum and online settings. As a result, we also provide simpler convergence analysis for R-SVRG and improve complexity bounds for R-SRG under finite-sum setting. Specifically, we prove that R-SRG achieves the same near-optimal complexity as R-SPIDER without requiring a small step size. Empirical experiments on a variety of tasks demonstrate effectiveness of proposed adaptive batch size scheme.