INTHOP: A Second-Order Globally Convergent Method for Nonconvex Optimization

TL;DR

INTHOP is a novel interval Hessian-based optimization algorithm that guarantees descent directions and global convergence, effectively addressing computational challenges in large-scale nonconvex problems.

Contribution

The paper introduces INTHOP, a second-order method that approximates the Hessian within intervals to ensure descent and reduce computational costs, with proven convergence.

Findings

INTHOP solves more problems with fewer evaluations than steepest descent and quasi-Newton methods.

It requires significantly less $O(n^3)$ operations than Newton's method for nonconvex problems.

The method demonstrates superior performance on extensive test problems.

Abstract

Second-order Newton-type algorithms that leverage the exact Hessian or its approximation are central to solve nonlinear optimization problems. However, their applications in solving large-scale nonconvex problems are hindered by three primary challenges: (1) the high computational cost associated with Hessian evaluations, (2) its inversion, and (3) ensuring descent direction at points where the Hessian becomes indefinite. We propose INTHOP, an interval Hessian-based optimization algorithm for nonconvex problems to deal with these primary challenges. The proposed search direction is based on approximating the original Hessian matrix by a positive definite matrix. The novelty of the proposed method is that the proposed search direction is guaranteed to be descent and requires approximation of Hessian and its inversion only at specific iterations. We prove that the difference between the calculated approximate and exact Hessian is bounded within an interval. Accordingly, the approximate Hessian matrix is reused if the iterates are in that chosen interval while computing the gradients at each iteration. We develop various algorithm variants based on the interval size updating methods and minimum eigenvalue computation methods. We also prove the global convergence of the proposed algorithm. Further, we apply the algorithm to an extensive set of test problems and compare its performance with the existing methods such as steepest descent, quasi-Newton, and Newton method. We show empirically that the proposed method solves more problems in fewer function and gradient evaluations than steepest descent and the quasi-Newton method. While in the comparison to the Newton method, we illustrate that for nonconvex optimization problems, we require substantially less $O(n^3)$ operations.