Complexity Guarantees for Nonconvex Newton-MR Under Inexact Hessian Information

TL;DR

This paper extends the Newton-MR algorithm for nonconvex optimization to cases with approximate Hessian information, providing convergence guarantees and complexity analysis under various conditions.

Contribution

It introduces a variant of Newton-MR with inexact Hessian handling, analyzing its convergence and complexity in nonconvex settings with theoretical guarantees.

Findings

Achieves global linear convergence under Polyak-ojasiewicz condition.

Converges to first-order sub-optimality at sub-linear rate in general nonconvex cases.

Performs competitively on machine learning problems compared to other methods.

Abstract

We consider an extension of the Newton-MR algorithm for nonconvex unconstrained optimization to the settings where Hessian information is approximated. Under a particular noise model on the Hessian matrix, we investigate the iteration and operation complexities of this variant to achieve appropriate sub-optimality criteria in several nonconvex settings. We do this by first considering functions that satisfy the (generalized) Polyak-\L ojasiewicz condition, a special sub-class of nonconvex functions. We show that, under certain conditions, our algorithm achieves global linear convergence rate. We then consider more general nonconvex settings where the rate to obtain first order sub-optimality is shown to be sub-linear. In all these settings, we show that our algorithm converges regardless of the degree of approximation of the Hessian as well as the accuracy of the solution to the sub-problem. Finally, we compare the performance of our algorithm with several alternatives on a few machine learning problems.