Linesearch Newton-CG methods for convex optimization with noise

TL;DR

This paper analyzes linesearch Newton-CG methods for convex optimization problems with noisy and inexact gradient, Hessian, and objective function evaluations, providing complexity bounds and focusing on machine learning applications.

Contribution

It introduces a comprehensive complexity analysis for inexact and noisy derivative evaluations in Newton-CG methods, including finite-sum minimization.

Findings

Expected iteration complexity bounds derived

Analysis includes bounded and dynamic accuracy noise models

Focus on applications in machine learning

Abstract

This paper studies the numerical solution of strictly convex unconstrained optimization problems by linesearch Newton-CG methods. We focus on methods employing inexact evaluations of the objective function and inexact and possibly random gradient and Hessian estimates. The derivative estimates are not required to satisfy suitable accuracy requirements at each iteration but with sufficiently high probability. Concerning the evaluation of the objective function we first assume that the noise in the objective function evaluations is bounded in absolute value. Then, we analyze the case where the error satisfies prescribed dynamic accuracy requirements. We provide for both cases a complexity analysis and derive expected iteration complexity bounds. We finally focus on the specific case of finite-sum minimization which is typical of machine learning applications.