Inertial Newton Algorithms Avoiding Strict Saddle Points

TL;DR

This paper investigates second-order algorithms combining Newton's method and inertial gradient descent, demonstrating they typically avoid strict saddle points in non-convex optimization and highlighting the influence of hyper-parameters.

Contribution

It provides theoretical analysis showing these algorithms almost always escape strict saddle points, supported by numerical experiments, and explores hyper-parameter effects.

Findings

Algorithms avoid strict saddle points in non-convex landscapes.

Hyper-parameters significantly influence behavior near critical points.

Numerical illustrations support theoretical results.

Abstract

We study the asymptotic behavior of second-order algorithms mixing Newton's method and inertial gradient descent in non-convex landscapes. We show that, despite the Newtonian behavior of these methods, they almost always escape strict saddle points. We also evidence the role played by the hyper-parameters of these methods in their qualitative behavior near critical points. The theoretical results are supported by numerical illustrations.