Advancing the lower bounds: An accelerated, stochastic, second-order method with optimal adaptation to inexactness

TL;DR

This paper introduces a new accelerated stochastic second-order method that is robust to inexact gradient and Hessian information, achieving optimal convergence and extending to higher-order derivatives.

Contribution

It develops a novel accelerated stochastic second-order algorithm with optimal convergence guarantees under inexactness, and generalizes to stochastic higher-order derivatives.

Findings

Achieves optimal convergence bounds in inexact gradient and Hessian settings.

Extends to stochastic higher-order derivatives with tensor generalization.

Matches the convergence of Nesterov Accelerated Tensor method in non-stochastic case.

Abstract

We present a new accelerated stochastic second-order method that is robust to both gradient and Hessian inexactness, which occurs typically in machine learning. We establish theoretical lower bounds and prove that our algorithm achieves optimal convergence in both gradient and Hessian inexactness in this key setting. We further introduce a tensor generalization for stochastic higher-order derivatives. When the oracles are non-stochastic, the proposed tensor algorithm matches the global convergence of Nesterov Accelerated Tensor method. Both algorithms allow for approximate solutions of their auxiliary subproblems with verifiable conditions on the accuracy of the solution.