A Newton-Based Method for Nonconvex Optimization with Fast Evasion of Saddle Points

TL;DR

This paper introduces a modified Newton-based method that efficiently escapes saddle points in nonconvex optimization problems, ensuring faster convergence to local minima in machine learning tasks.

Contribution

It proposes a novel second-order algorithm that replaces negative Hessian eigenvalues with their absolute values, enabling rapid saddle point escape.

Findings

Escapes saddle points in at most logarithmic iterations

Converges to a local minimum with high probability

Operates independently of problem-specific constants

Abstract

Machine learning problems such as neural network training, tensor decomposition, and matrix factorization, require local minimization of a nonconvex function. This local minimization is challenged by the presence of saddle points, of which there can be many and from which descent methods may take inordinately large number of iterations to escape. This paper presents a second-order method that modifies the update of Newton's method by replacing the negative eigenvalues of the Hessian by their absolute values and uses a truncated version of the resulting matrix to account for the objective's curvature. The method is shown to escape saddles in at most $1 + \log_{3/2} (\delta/2\varepsilon)$ iterations where $\varepsilon$ is the target optimality and $\delta$ characterizes a point sufficiently far away from the saddle. This base of this exponential escape is $3/2$ independently of problem constants. Adding classical properties of Newton's method, the paper proves convergence to a local minimum with probability $1-p$ in $O\left(\log(1/p)) + O(\log(1/\varepsilon)\right)$ iterations.