Stochastic Non-convex Optimization with Strong High Probability Second-order Convergence

TL;DR

This paper introduces novel stochastic algorithms for non-convex optimization that achieve high probability second-order convergence with near-linear time complexity, advancing the theoretical understanding of stochastic non-convex optimization.

Contribution

The paper proposes the NCG-S step and two algorithms that are the first to attain high probability second-order convergence with near-linear complexity.

Findings

First stochastic algorithms with high probability second-order convergence.

Algorithms have almost linear time complexity in problem dimension.

Significantly advances theoretical guarantees in stochastic non-convex optimization.

Abstract

In this paper, we study stochastic non-convex optimization with non-convex random functions. Recent studies on non-convex optimization revolve around establishing second-order convergence, i.e., converging to a nearly second-order optimal stationary points. However, existing results on stochastic non-convex optimization are limited, especially with a high probability second-order convergence. We propose a novel updating step (named NCG-S) by leveraging a stochastic gradient and a noisy negative curvature of a stochastic Hessian, where the stochastic gradient and Hessian are based on a proper mini-batch of random functions. Building on this step, we develop two algorithms and establish their high probability second-order convergence. To the best of our knowledge, the proposed stochastic algorithms are the first with a second-order convergence in {\it high probability} and a time complexity that is {\it almost linear} in the problem's dimensionality.