Neon2: Finding Local Minima via First-Order Oracles

TL;DR

Neon2 introduces a first-order reduction technique for non-convex optimization that transforms stationary-point algorithms into local-minimum finders, eliminating the need for Hessian-vector products while maintaining performance.

Contribution

It provides a novel reduction that converts existing algorithms into local-minimum finders using only gradient computations, applicable in stochastic and deterministic settings.

Findings

Transforms Natasha2 into a first-order method without performance loss

Converts SGD, GD, SCSG, SVRG into local-minimum finding algorithms

Outperforms some of the best known results in local minima finding

Abstract

We propose a reduction for non-convex optimization that can (1) turn an stationary-point finding algorithm into an local-minimum finding one, and (2) replace the Hessian-vector product computations with only gradient computations. It works both in the stochastic and the deterministic settings, without hurting the algorithm's performance. As applications, our reduction turns Natasha2 into a first-order method without hurting its performance. It also converts SGD, GD, SCSG, and SVRG into algorithms finding approximate local minima, outperforming some best known results.