Convergence of Newton-MR under Inexact Hessian Information

TL;DR

This paper analyzes the convergence of Newton-MR with inexact Hessian information, extending classical Newton-CG methods to invex problems and demonstrating robustness and efficiency in large-scale applications.

Contribution

It introduces a convergence analysis for Newton-MR with approximate Hessians using matrix perturbation theory, broadening its applicability beyond convex problems.

Findings

Newton-MR converges reliably with inexact Hessians.

The method shows high efficiency in large-scale problems.

Numerical experiments confirm robustness to Hessian approximations.

Abstract

Recently, there has been a surge of interest in designing variants of the classical Newton-CG in which the Hessian of a (strongly) convex function is replaced by suitable approximations. This is mainly motivated by large-scale finite-sum minimization problems that arise in many machine learning applications. Going beyond convexity, inexact Hessian information has also been recently considered in the context of algorithms such as trust-region or (adaptive) cubic regularization for general non-convex problems. Here, we do that for Newton-MR, which extends the application range of the classical Newton-CG beyond convexity to invex problems. Unlike the convergence analysis of Newton-CG, which relies on spectrum preserving Hessian approximations in the sense of L\"{o}wner partial order, our work here draws from matrix perturbation theory to estimate the distance between the subspaces underlying the exact and approximate Hessian matrices. Numerical experiments demonstrate a great degree of resilience to such Hessian approximations, amounting to a highly efficient algorithm in large-scale problems.