Some iterative algorithms on Riemannian manifolds and Banach spaces with good global convergence guarantee

TL;DR

This paper introduces new iterative optimization algorithms on Riemannian manifolds and Banach spaces with strong global convergence guarantees, avoiding saddle points and applicable under broad conditions.

Contribution

The paper develops novel iterative algorithms with proven global convergence on manifolds and Banach spaces, extending optimization theory to infinite-dimensional and curved spaces.

Findings

Algorithms converge to critical points or diverge to infinity.

Sequences avoid saddle points with random initialization.

Applicable to functions with broad regularity and critical point conditions.

Abstract

In this paper, we introduce some new iterative optimisation algorithms on Riemannian manifolds and Hilbert spaces which have good global convergence guarantees to local minima. More precisely, these algorithms have the following properties: If $\{x_n\}$ is a sequence constructed by one such algorithm then: - Finding critical points: Any cluster point of $\{x_n\}$ is a critical point of the cost function $f$. - Convergence guarantee: Under suitable assumptions, the sequence $\{x_n\}$ either converges to a point $x^*$, or diverges to $\infty$. - Avoidance of saddle points: If $x_0$ is randomly chosen, then the sequence $\{x_n\}$ cannot converge to a saddle point. Our results apply for quite general situations: the cost function $f$ is assumed to be only $C^2$ or $C^3$, and either $f$ has at most countably many critical points (which is a generic situation) or satisfies certain Lojasiewicz gradient inequalities. To illustrate the results, we provide a nice application with optimisation over the unit sphere in a Euclidean space. As for tools needed for the results, in the Riemannian manifold case we introduce a notion of "strong local retraction" and (to deal with Newton's method type) a notion of "real analytic-like strong local retraction". In the case of Banach spaces, we introduce a slight generalisation of the notion of "shyness", and design a new variant of Backtracking New Q-Newton's method which is more suitable to the infinite dimensional setting (and in the Euclidean setting is simpler than the current versions).