Newton-Type Methods for Non-Convex Optimization Under Inexact Hessian Information

TL;DR

This paper develops trust-region and cubic regularization methods for non-convex optimization that work with inexact Hessians, using relaxed conditions and sampling strategies to achieve optimal iteration complexity.

Contribution

It introduces mild conditions for inexact Hessian approximations, enabling the use of random sampling methods in non-convex optimization algorithms.

Findings

Relaxed Hessian approximation conditions are sufficient for convergence.

Sampling strategies can construct Hessians with optimal complexity.

Methods achieve tight iteration complexity bounds for non-convex problems.

Abstract

We consider variants of trust-region and cubic regularization methods for non-convex optimization, in which the Hessian matrix is approximated. Under mild conditions on the inexact Hessian, and using approximate solution of the corresponding sub-problems, we provide iteration complexity to achieve $ \epsilon $-approximate second-order optimality which have shown to be tight. Our Hessian approximation conditions constitute a major relaxation over the existing ones in the literature. Consequently, we are able to show that such mild conditions allow for the construction of the approximate Hessian through various random sampling methods. In this light, we consider the canonical problem of finite-sum minimization, provide appropriate uniform and non-uniform sub-sampling strategies to construct such Hessian approximations, and obtain optimal iteration complexity for the corresponding sub-sampled trust-region and cubic regularization methods.