A Regularized Hessian-Free Inexact Newton-Type Method with Global $\mathcal{O}(k^{-2})$ Convergence

TL;DR

This paper introduces a regularized Hessian-free Newton method with proven global convergence rate of (k^{-2}) for smooth convex functions, combining theoretical rigor with practical efficiency.

Contribution

It develops a novel adaptive regularized Hessian-free Newton method that achieves optimal convergence rates and outperforms existing methods in computational experiments.

Findings

Achieves (k^{-2}) convergence rate for smooth convex optimization.

Incorporates an adaptive regularization parameter for efficiency.

Outperforms the Regularized Newton Method in benchmark tests.

Abstract

We propose a regularized Hessian-free Newton-type method for minimizing smooth convex functions with Lipschitz continuous Hessians. The algorithm constructs an approximate Hessian by finite differences and selects the regularization parameter through an adaptive criterion that ensures sufficient decrease and gradient control. We prove that the method achieves a global $\mathcal{O}(k^{-2})$ convergence rate, matching the best known bound for second-order methods. A modified variant incorporating the exact Hessian when available enjoys local quadratic convergence under standard assumptions. Despite its simplicity, this variant is computationally faster than the \emph{Regularized Newton Method} of Mishchenko (2023) across several convex benchmark problems. Our analysis also provides explicit bounds on the regularization sequence and a worst-case iteration complexity of order $\mathcal{O}(\varepsilon^{-2})$. The proposed framework thus unifies regularized and Hessian-free Newton-type schemes, offering a theoretically sound and practically efficient alternative for smooth convex optimization.