Natasha 2: Faster Non-Convex Optimization Than SGD

TL;DR

This paper introduces Natasha 2, a stochastic algorithm that significantly accelerates finding approximate local minima in smooth neural networks and nonconvex functions compared to traditional SGD, using fewer backpropagations.

Contribution

The paper presents Natasha 2, a novel stochastic algorithm that achieves faster convergence rates for non-convex optimization than SGD, with theoretical guarantees.

Findings

Achieves $O( ext{epsilon}^{-3.25})$ complexity for local minima

Outperforms SGD's $O( ext{epsilon}^{-4})$ complexity

Applicable to any smooth neural network and nonconvex function

Abstract

We design a stochastic algorithm to train any smooth neural network to $\varepsilon$-approximate local minima, using $O(\varepsilon^{-3.25})$ backpropagations. The best result was essentially $O(\varepsilon^{-4})$ by SGD. More broadly, it finds $\varepsilon$-approximate local minima of any smooth nonconvex function in rate $O(\varepsilon^{-3.25})$, with only oracle access to stochastic gradients.