Riemannian Stochastic Gradient Method for Nested Composition Optimization

TL;DR

This paper introduces a Riemannian stochastic gradient method for nested composition optimization problems on manifolds, addressing bias issues and providing convergence guarantees with applications in reinforcement learning.

Contribution

It proposes the R-SCGD algorithm for two-level and multi-level nested compositional problems on Riemannian manifolds, with proven convergence rates.

Findings

Converges to an approximate stationary point in $O( ext{epsilon}^{-2})$ calls.

Generalizes to multi-level nested structures with same complexity.

Numerical evaluation demonstrates effectiveness in reinforcement learning.

Abstract

This work considers optimization of composition of functions in a nested form over Riemannian manifolds where each function contains an expectation. This type of problems is gaining popularity in applications such as policy evaluation in reinforcement learning or model customization in meta-learning. The standard Riemannian stochastic gradient methods for non-compositional optimization cannot be directly applied as stochastic approximation of inner functions create bias in the gradients of the outer functions. For two-level composition optimization, we present a Riemannian Stochastic Composition Gradient Descent (R-SCGD) method that finds an approximate stationary point, with expected squared Riemannian gradient smaller than $\epsilon$, in $O(\epsilon^{-2})$ calls to the stochastic gradient oracle of the outer function and stochastic function and gradient oracles of the inner function. Furthermore, we generalize the R-SCGD algorithms for problems with multi-level nested compositional structures, with the same complexity of $O(\epsilon^{-2})$ for the first-order stochastic oracle. Finally, the performance of the R-SCGD method is numerically evaluated over a policy evaluation problem in reinforcement learning.