Newton-MR: Inexact Newton Method With Minimum Residual Sub-problem Solver

TL;DR

Newton-MR introduces an inexact Newton method using Minimum Residual sub-problem solving, applicable to non-convex invex problems, with proven global and local convergence under weaker conditions, and demonstrates competitive performance in machine learning tasks.

Contribution

It proposes a novel inexact Newton method, Newton-MR, with weaker convergence assumptions and broad applicability to invex problems, including non-convex cases.

Findings

Global convergence under joint regularity conditions

Local convergence to minima set

Competitive numerical performance in machine learning

Abstract

We consider a variant of inexact Newton Method, called Newton-MR, in which the least-squares sub-problems are solved approximately using Minimum Residual method. By construction, Newton-MR can be readily applied for unconstrained optimization of a class of non-convex problems known as invex, which subsumes convexity as a sub-class. For invex optimization, instead of the classical Lipschitz continuity assumptions on gradient and Hessian, Newton-MR's global convergence can be guaranteed under a weaker notion of joint regularity of Hessian and gradient. We also obtain Newton-MR's problem-independent local convergence to the set of minima. We show that fast local/global convergence can be guaranteed under a novel inexactness condition, which, to our knowledge, is much weaker than the prior related works. Numerical results demonstrate the performance of Newton-MR as compared with several other Newton-type alternatives on a few machine learning problems.