Stochastic Newton Proximal Extragradient Method

TL;DR

This paper introduces a stochastic Newton proximal extragradient method that enhances convergence rates for strongly convex optimization, combining fast global linear and superlinear local convergence using noisy Hessian estimates.

Contribution

It extends the HPE framework to achieve improved global and local convergence rates for stochastic second-order methods with noisy Hessian oracles.

Findings

Achieves faster global linear convergence rate.

Reaches superlinear convergence in fewer iterations.

Improves upon previous Hessian averaging methods.

Abstract

Stochastic second-order methods achieve fast local convergence in strongly convex optimization by using noisy Hessian estimates to precondition the gradient. However, these methods typically reach superlinear convergence only when the stochastic Hessian noise diminishes, increasing per-iteration costs over time. Recent work in [arXiv:2204.09266] addressed this with a Hessian averaging scheme that achieves superlinear convergence without higher per-iteration costs. Nonetheless, the method has slow global convergence, requiring up to $\tilde{O}(\kappa^2)$ iterations to reach the superlinear rate of $\tilde{O}((1/t)^{t/2})$, where $\kappa$ is the problem's condition number. In this paper, we propose a novel stochastic Newton proximal extragradient method that improves these bounds, achieving a faster global linear rate and reaching the same fast superlinear rate in $\tilde{O}(\kappa)$ iterations. We accomplish this by extending the Hybrid Proximal Extragradient (HPE) framework, achieving fast global and local convergence rates for strongly convex functions with access to a noisy Hessian oracle.