A Novel Convergence Analysis for Algorithms of the Adam Family

TL;DR

This paper provides a simple, generic convergence analysis for Adam-style algorithms, covering many variants and broad non-convex problems, under mild assumptions on stochastic gradients and momentum parameters.

Contribution

It introduces a unified convergence proof for Adam and its variants, applicable to complex non-convex optimization problems with minimal assumptions.

Findings

Convergence holds for Adam, AMSGrad, Adabound under mild conditions.

Analysis requires only a large momentum parameter and bounded adaptive step size.

Applicable to non-convex problems like min-max, compositional, bilevel optimization.

Abstract

Since its invention in 2014, the Adam optimizer has received tremendous attention. On one hand, it has been widely used in deep learning and many variants have been proposed, while on the other hand their theoretical convergence property remains to be a mystery. It is far from satisfactory in the sense that some studies require strong assumptions about the updates, which are not necessarily applicable in practice, while other studies still follow the original problematic convergence analysis of Adam, which was shown to be not sufficient to ensure convergence. Although rigorous convergence analysis exists for Adam, they impose specific requirements on the update of the adaptive step size, which are not generic enough to cover many other variants of Adam. To address theses issues, in this extended abstract, we present a simple and generic proof of convergence for a family of Adam-style methods (including Adam, AMSGrad, Adabound, etc.). Our analysis only requires an increasing or large "momentum" parameter for the first-order moment, which is indeed the case used in practice, and a boundness condition on the adaptive factor of the step size, which applies to all variants of Adam under mild conditions of stochastic gradients. We also establish a variance diminishing result for the used stochastic gradient estimators. Indeed, our analysis of Adam is so simple and generic that it can be leveraged to establish the convergence for solving a broader family of non-convex optimization problems, including min-max, compositional, and bilevel optimization problems. For the full (earlier) version of this extended abstract, please refer to arXiv:2104.14840.