An Invitation to Higher-Order Riemannian Optimization: Optimal and Implementable Methods

TL;DR

This paper introduces optimal higher-order Riemannian optimization methods that match Euclidean complexity, along with a new framework for analyzing higher-order regularity on manifolds, including practical algorithms for cubic regularization.

Contribution

It develops the first optimal-rate higher-order methods for non-convex Riemannian optimization and introduces a novel framework using pullback connections and the Sasaki metric.

Findings

Optimal oracle complexity matches Euclidean case.

Framework for higher-order regularity on manifolds.

First Krylov-based method for quartically regularized cubic polynomials.

Abstract

This paper presents the first optimal-rate $p$-th order methods with $p\geq 1$ for finding first and second-order stationary points of non-convex smooth objective functions over Riemannian manifolds. In contrast to the geodesically convex setting, we definitively establish that the optimal oracle complexity of non-convex optimization over manifolds matches that over Euclidean space. In parallel with the complexity analysis, we introduce a general framework for systematically studying higher-order regularity on Riemannian manifolds that characterizes its joint dependence on the objective function and the chosen retraction. To the best of our knowledge, this framework constitutes the first known application in optimization of pullback connections and the Sasaki metric to the study of retraction-based pullbacks of the objective function. We provide clean derivative bounds based on a new covariant Fa\`a di Bruno formula derived within our framework. For $p=3$, our methods are fully implementable via a new Krylov-based framework for minimizing quartically regularized cubic polynomials. This is the first Krylov method for this class of polynomials and may be of independent interest beyond Riemannian optimization.