Regularized Newton Method with Global $O(1/k^2)$ Convergence

TL;DR

This paper introduces a globally convergent Newton-type optimization method that combines cubic regularization and adaptive Levenberg--Marquardt penalty, achieving an optimal $O(1/k^2)$ convergence rate with cheap iterations.

Contribution

It proposes the first Newton variant with both low-cost iterations and guaranteed fast global convergence for convex objectives with Lipschitz Hessians.

Findings

Achieves $O(1/k^2)$ convergence rate globally.

Ensures superlinear local convergence for strongly convex objectives.

Includes an adaptive line search that does not require prior knowledge of parameters.

Abstract

We present a Newton-type method that converges fast from any initialization and for arbitrary convex objectives with Lipschitz Hessians. We achieve this by merging the ideas of cubic regularization with a certain adaptive Levenberg--Marquardt penalty. In particular, we show that the iterates given by $x^{k+1}=x^k - \bigl(\nabla^2 f(x^k) + \sqrt{H\|\nabla f(x^k)\|} \mathbf{I}\bigr)^{-1}\nabla f(x^k)$, where $H>0$ is a constant, converge globally with a $\mathcal{O}(\frac{1}{k^2})$ rate. Our method is the first variant of Newton's method that has both cheap iterations and provably fast global convergence. Moreover, we prove that locally our method converges superlinearly when the objective is strongly convex. To boost the method's performance, we present a line search procedure that does not need prior knowledge of $H$ and is provably efficient.