Learning-Rate-Free Stochastic Optimization over Riemannian Manifolds

TL;DR

This paper introduces novel learning-rate-free stochastic optimization algorithms for Riemannian manifolds, removing the need for hyperparameter tuning and achieving near-optimal convergence guarantees validated by numerical experiments.

Contribution

It presents the first learning-rate-free algorithms for stochastic Riemannian optimization with proven convergence guarantees and practical competitive performance.

Findings

Achieves high probability convergence guarantees close to the optimal tuned rates.

Demonstrates competitive empirical performance against traditional algorithms requiring tuning.

Eliminates the need for hyperparameter tuning in Riemannian stochastic optimization.

Abstract

In recent years, interest in gradient-based optimization over Riemannian manifolds has surged. However, a significant challenge lies in the reliance on hyperparameters, especially the learning rate, which requires meticulous tuning by practitioners to ensure convergence at a suitable rate. In this work, we introduce innovative learning-rate-free algorithms for stochastic optimization over Riemannian manifolds, eliminating the need for hand-tuning and providing a more robust and user-friendly approach. We establish high probability convergence guarantees that are optimal, up to logarithmic factors, compared to the best-known optimally tuned rate in the deterministic setting. Our approach is validated through numerical experiments, demonstrating competitive performance against learning-rate-dependent algorithms.