Riemannian Stochastic Fixed Point Optimization Algorithm

TL;DR

This paper introduces a novel Riemannian stochastic fixed point optimization algorithm that combines fixed point approximation with adaptive learning rates, providing convergence guarantees for complex hierarchical optimization problems on manifolds.

Contribution

It proposes a new algorithm integrating fixed point methods on Riemannian manifolds with adaptive learning rates, and provides convergence analysis for both convex and nonconvex cases.

Findings

The algorithm effectively approximates solutions with small constant step-sizes.

Guaranteed convergence is achieved with diminishing step-size sequences.

Numerical experiments show the algorithm's effectiveness with Adam and AMSGrad formulas.

Abstract

This paper considers a stochastic optimization problem over the fixed point sets of quasinonexpansive mappings on Riemannian manifolds. The problem enables us to consider Riemannian hierarchical optimization problems over complicated sets, such as the intersection of many closed convex sets, the set of all minimizers of a nonsmooth convex function, and the intersection of sublevel sets of nonsmooth convex functions. We focus on adaptive learning rate optimization algorithms, which adapt step-sizes (referred to as learning rates in the machine learning field) to find optimal solutions quickly. We then propose a Riemannian stochastic fixed point optimization algorithm, which combines fixed point approximation methods on Riemannian manifolds with the adaptive learning rate optimization algorithms. We also give convergence analyses of the proposed algorithm for nonsmooth convex and smooth nonconvex optimization. The analysis results indicate that, with small constant step-sizes, the proposed algorithm approximates a solution to the problem. Consideration of the case in which step-size sequences are diminishing demonstrates that the proposed algorithm solves the problem with a guaranteed convergence rate. This paper also provides numerical comparisons that demonstrate the effectiveness of the proposed algorithms with formulas based on the adaptive learning rate optimization algorithms, such as Adam and AMSGrad.