OFF (Proximal) Newton-type Methods with Inexact Derivatives for Unconstrained Optimization

TL;DR

This paper introduces OFF variants of proximal and regularized Newton methods that use inexact derivatives for nonconvex optimization, maintaining convergence rates and enabling practical, verifiable accuracy strategies.

Contribution

It develops objective-function-free Newton algorithms with inexact derivatives, providing convergence guarantees and practical schemes for nonconvex optimization.

Findings

Algorithms achieve superlinear and quadratic convergence in expectation.

The methods maintain convergence rates with inexact derivatives.

Complexity matches optimal results in the literature.

Abstract

In this paper, we propose objective-function-free (OFF) variants of the proximal Newton method for nonconvex composite optimization problems and the regularized Newton method for unconstrained optimization problems, respectively, using inexact gradients and Hessians. Theoretical analyses verify that the global and local convergence rates of the proposed algorithms remain consistent with those attained under exact evaluations of the objective function and derivatives. To validate the practical applicability of the theoretical assumptions, a lazy gradient strategy is adopted to provide a verifiable scheme for satisfying the accuracy criteria of approximate gradients. For finite-sum optimization problems, an adaptive sampling strategy is further developed to eliminate the circular dependency between sample size and gradient estimation. The proposed algorithm is proven to achieve locally superlinear and quadratic convergence in expectation. Furthermore, we present an OFF regularized Newton and negative curvature algorithm, which uses inexact derivatives to locate approximate second-order stationary points in unconstrained optimization. The worst-case iteration and (sample) operation complexity of the developed algorithm is consistent with the optimal results in the existing literature.