A Regularized Newton Method for Nonconvex Optimization with Global and Local Complexity Guarantees

TL;DR

This paper introduces an adaptive regularized Newton method for nonconvex optimization that achieves both optimal global complexity and quadratic local convergence without prior knowledge of the Hessian Lipschitz constant.

Contribution

It proposes a new class of regularizers and a conjugate gradient approach with a negative curvature monitor, bridging the gap between global and local convergence guarantees.

Findings

Achieves $O( ext{epsilon}^{-3/2})$ global complexity for second-order oracle calls.

Attains $ ilde{O}( ext{epsilon}^{-7/4})$ complexity for Hessian-vector products.

Exhibits quadratic local convergence near positive definite Hessian points.

Abstract

Finding an $\epsilon$-stationary point of a nonconvex function with a Lipschitz continuous Hessian is a central problem in optimization. Regularized Newton methods are a classical tool and have been studied extensively, yet they still face a trade-off between global and local convergence. Whether a parameter-free algorithm of this type can simultaneously achieve optimal global complexity and quadratic local convergence remains an open question. To bridge this long-standing gap, we propose a new class of regularizers constructed from the current and previous gradients, and leverage the conjugate gradient approach with a negative curvature monitor to solve the regularized Newton equation. The proposed algorithm is adaptive, requiring no prior knowledge of the Hessian Lipschitz constant, and achieves a global complexity of $O(\epsilon^{-3/2})$ in terms of the second-order oracle calls, and $\tilde{O}(\epsilon^{-7/4})$ for Hessian-vector products, respectively. When the iterates converge to a point where the Hessian is positive definite, the method exhibits quadratic local convergence. Preliminary numerical results, including training the physics-informed neural networks, illustrate the competitiveness of our algorithm.