First- and Second-Order Stochastic Adaptive Regularization with Cubics: High Probability Iteration and Sample Complexity

TL;DR

This paper establishes high-probability iteration and sample complexity bounds for stochastic adaptive regularization with cubics, improving understanding of their efficiency in finding first- and second-order stationary points.

Contribution

It provides the first high-probability complexity bounds for stochastic adaptive regularization methods with cubics targeting both first- and second-order optimality.

Findings

High-probability bounds outperform other stochastic methods.

Applicable to various optimization settings including risk minimization.

Demonstrates efficiency of SARC methods in stochastic settings.

Abstract

We present high-probability (and expectation) complexity bounds for two versions of stochastic adaptive regularization methods with cubics (SARC), also known as regularized Newton methods. The first algorithm aims to find first-order stationary points, while the second targets second-order optimality conditions. Both methods employ stochastic zeroth-, first-, and second-order oracles with specific accuracy and reliability requirements. These oracles, which have been previously used with other stochastic adaptive methods like trust-region and line-search algorithms, are applicable to various optimization settings including expected risk minimization and simulation optimization. In this paper, we establish the first high-probability iteration and sample complexity bounds for both first- and second-order SARC algorithms. Our analysis demonstrates that as in the deterministic case, they outperform other stochastic adaptive methods.