Riemannian Adaptive Regularized Newton Methods with H\"older Continuous Hessians

TL;DR

This paper establishes new worst-case complexity bounds for Riemannian adaptive regularized Newton methods, accounting for Hölder continuous Hessians, retraction smoothness, and inexact subproblem solutions, unifying and extending existing optimization frameworks.

Contribution

It provides the first sharp complexity guarantees for Riemannian adaptive regularized Newton methods with Hölder continuous Hessians, encompassing both RAR and RTR methods.

Findings

Complexity bounds depend on Hölder continuity parameters.

Methods locate approximate second-order stationary points efficiently.

Results unify and extend prior complexity analyses.

Abstract

This paper presents strong worst-case iteration and operation complexity guarantees for Riemannian adaptive regularized Newton methods, a unified framework encompassing both Riemannian adaptive regularization (RAR) methods and Riemannian trust region (RTR) methods. We comprehensively characterize the sources of approximation in second-order manifold optimization methods: the objective function's smoothness, retraction's smoothness, and subproblem solver's inexactness. Specifically, for a function with a $\mu$-H\"older continuous Hessian, when equipped with a retraction featuring a $\nu$-H\"older continuous differential and a $\theta$-inexact subproblem solver, both RTR and RAR with $2+\alpha$ regularization (where $\alpha=\min\{\mu,\nu,\theta\}$) locate an $(\epsilon,\epsilon^{\alpha/(1+\alpha)})$-approximate second-order stationary point within at most $O(\epsilon^{-(2+\alpha)/(1+\alpha)})$ iterations and at most $\tilde{O}(\epsilon^{-(4+3\alpha)/(2(1+\alpha))})$ Hessian-vector products. These complexity results are novel and sharp, and reduce to an iteration complexity of $O(\epsilon^{-3/2})$ and an operation complexity of $\tilde{O}(\epsilon^{-7/4})$ when $\alpha=1$.