A MINRES-based Linesearch Algorithm for Nonconvex Optimization with Non-positive Curvature Detection

TL;DR

This paper introduces a MINRES-based Newton algorithm for nonconvex optimization that effectively detects non-positive curvature, avoids saddle points, and converges rapidly to minima, with proven theoretical guarantees and demonstrated numerical efficiency.

Contribution

The paper presents a novel MINRES-based method integrating curvature detection and linesearch for nonconvex problems, with convergence analysis and practical performance validation.

Findings

Algorithm avoids strict saddle points under certain conditions.

Proven superlinear convergence near minima with Polyak-Łojasiewicz condition.

Numerical experiments show efficiency on benchmark and deep learning problems.

Abstract

We propose a MINRES-based Newton-type algorithm for solving unconstrained nonconvex optimization problems. Our approach uses the minimal residual method (MINRES), a well-known solver for indefinite symmetric linear systems, to compute descent directions that leverage second-order and non-positive curvature (NPC) information. Comprehensive asymptotic convergence properties are derived under standard assumptions. In particular, under the Kurdyka-{\L}ojasiewicz inequality and a mild NPC-detectability condition, we prove that our algorithm can avoid strict saddle points and converge to second-order critical points. This is primarily achieved by integrating proper regularization techniques and forward linesearch mechanisms along NPC directions. Furthermore, fast local superlinear convergence to potentially non-isolated minima is established, when the local Polyak-{\L}ojasiewicz condition is satisfied. Numerical experiments on the CUTEst test collection and on a deep auto-encoder problem illustrate the efficiency of the proposed method.