Gradient descent avoids strict saddles with a simple line-search method too

TL;DR

This paper proves that a modified line-search gradient descent method can avoid strict saddle points on smooth functions, extending the guarantee to Riemannian manifolds and relaxing common assumptions.

Contribution

It introduces a new convergence guarantee for line-search gradient descent avoiding strict saddles, using the Luzin N^{-1} property and extending to Riemannian optimization.

Findings

Line-search GD avoids strict saddles on $C^2$ functions.

Extension of guarantees to Riemannian gradient descent.

Improved convergence guarantees for RGD with constant step size.

Abstract

It is known that gradient descent (GD) on a $C^2$ cost function generically avoids strict saddle points when using a small, constant step size. However, no such guarantee existed for GD with a line-search method. We provide one for a modified version of the standard Armijo backtracking method with generic, arbitrarily large initial step size. The proof underlines the double role of the Luzin $N^{-1}$ property for the iteration maps, and allows to forgo the habitual Lipschitz gradient assumption. We extend this to the Riemannian setting (RGD), assuming the retraction is real analytic (though the cost function still only needs to be $C^2$). In closing, we also improve guarantees for RGD with a constant step size in some scenarios.