On Relatively Smooth Optimization over Riemannian Manifolds

TL;DR

This paper introduces two novel Riemannian first-order optimization methods that incorporate Bregman distances, enabling efficient handling of relatively smooth problems on manifolds with theoretical convergence guarantees.

Contribution

The paper develops retraction-based and projection-based Riemannian Bregman gradient methods, extending optimization techniques to relatively smooth functions on manifolds with convergence analysis.

Findings

Achieve $ ext{O}(1/ ext{epsilon}^2)$ iteration complexity for stationary points.

Develop stochastic variants with $ ext{O}(1/ ext{epsilon}^4)$ sample complexity on compact manifolds.

Numerical experiments demonstrate the effectiveness of the proposed methods.

Abstract

We study optimization over Riemannian embedded submanifolds, where the objective function is relatively smooth in the ambient Euclidean space. Such problems have broad applications but are still largely unexplored. We introduce two Riemannian first-order methods, namely the retraction-based and projection-based Riemannian Bregman gradient methods, by incorporating the Bregman distance into the update steps. The retraction-based method can handle nonsmooth optimization; at each iteration, the update direction is generated by solving a convex optimization subproblem constrained to the tangent space. We show that when the reference function is of the quartic form $h(x) = \frac{1}{4}\|x\|^4 + \frac{1}{2}\|x\|^2$, the constraint subproblem admits a closed-form solution. The projection-based approach can be applied to smooth Riemannian optimization, which solves an unconstrained subproblem in the ambient Euclidean space. Both methods are shown to achieve an iteration complexity of $\mathcal{O}(1/\epsilon^2)$ for finding an $\epsilon$-approximate Riemannian stationary point. When the manifold is compact, we further develop stochastic variants and establish a sample complexity of $\mathcal{O}(1/\epsilon^4)$. Numerical experiments on the nonlinear eigenvalue problem and low-rank quadratic sensing problem demonstrate the advantages of the proposed methods.