Accelerated Methods for Non-Convex Optimization

TL;DR

This paper introduces a Hessian-free accelerated gradient method for non-convex optimization that achieves faster convergence to stationary points with second-order guarantees, suitable for large-scale problems.

Contribution

It proposes a novel accelerated method with improved complexity for non-convex optimization, requiring only gradient evaluations and providing second-order guarantees.

Findings

Achieves $O( ext{epsilon}^{-7/4} ext{log}(1/ ext{epsilon}))$ complexity

Provides second-order guarantees with Hessian-free approach

Suitable for large-scale non-convex problems

Abstract

We present an accelerated gradient method for non-convex optimization problems with Lipschitz continuous first and second derivatives. The method requires time $O(\epsilon^{-7/4} \log(1/ \epsilon) )$ to find an $\epsilon$-stationary point, meaning a point $x$ such that $\|\nabla f(x)\| \le \epsilon$. The method improves upon the $O(\epsilon^{-2} )$ complexity of gradient descent and provides the additional second-order guarantee that $\nabla^2 f(x) \succeq -O(\epsilon^{1/2})I$ for the computed $x$. Furthermore, our method is Hessian free, i.e. it only requires gradient computations, and is therefore suitable for large scale applications.