A general framework of Riemannian adaptive optimization methods with a convergence analysis

TL;DR

This paper introduces a comprehensive Riemannian adaptive optimization framework, including AMSGrad on submanifolds, with convergence analysis and applications to PCA and matrix completion.

Contribution

It presents a unified framework for Riemannian adaptive methods, including AMSGrad, with convergence guarantees and practical applications.

Findings

Convergence rates depend on mini-batch size.

The framework encompasses several existing algorithms.

Numerical experiments validate the effectiveness on PCA and matrix completion.

Abstract

This paper proposes a general framework of Riemannian adaptive optimization methods. The framework encapsulates several stochastic optimization algorithms on Riemannian manifolds and incorporates the mini-batch strategy that is often used in deep learning. Within this framework, we also propose AMSGrad on embedded submanifolds of Euclidean space. Moreover, we give convergence analyses valid for both a constant and a diminishing step size. Our analyses also reveal the relationship between the convergence rate and mini-batch size. In numerical experiments, we applied the proposed algorithm to principal component analysis and the low-rank matrix completion problem, which can be considered to be Riemannian optimization problems. Python implementations of the methods used in the numerical experiments are available at https://github.com/iiduka-researches/202408-adaptive.