Nesterov acceleration in benignly non-convex landscapes

TL;DR

This paper extends the theoretical understanding of Nesterov acceleration to benignly non-convex landscapes common in deep learning, showing similar guarantees as in convex settings, including stochastic variants.

Contribution

It demonstrates that Nesterov acceleration guarantees hold in benign non-convex landscapes, bridging the gap between theory and deep learning practice.

Findings

Nesterov acceleration achieves similar guarantees in benign non-convex landscapes as in convex cases.

The results apply to both continuous and discrete time models of NAG.

Stochastic NAG with additive and multiplicative noise also benefits from these guarantees.

Abstract

While momentum-based optimization algorithms are commonly used in the notoriously non-convex optimization problems of deep learning, their analysis has historically been restricted to the convex and strongly convex setting. In this article, we partially close this gap between theory and practice and demonstrate that virtually identical guarantees can be obtained in optimization problems with a `benign' non-convexity. We show that these weaker geometric assumptions are well justified in overparametrized deep learning, at least locally. Variations of this result are obtained for a continuous time model of Nesterov's accelerated gradient descent algorithm (NAG), the classical discrete time version of NAG, and versions of NAG with stochastic gradient estimates with purely additive noise and with noise that exhibits both additive and multiplicative scaling.