A Simple Convergence Proof of Adam and Adagrad

TL;DR

This paper presents a straightforward convergence proof for Adam and Adagrad algorithms on smooth, possibly non-convex functions, providing explicit bounds and insights into their convergence behaviors and parameter dependencies.

Contribution

It offers the first simple convergence proof for Adam and Adagrad on non-convex functions with explicit bounds and improved dependency on momentum decay rate.

Findings

Adam converges with the same rate as Adagrad under proper hyper-parameters.

Default Adam parameters do not guarantee convergence but still move away faster than Adagrad.

The convergence bound's dependency on the momentum decay rate is significantly improved.

Abstract

We provide a simple proof of convergence covering both the Adam and Adagrad adaptive optimization algorithms when applied to smooth (possibly non-convex) objective functions with bounded gradients. We show that in expectation, the squared norm of the objective gradient averaged over the trajectory has an upper-bound which is explicit in the constants of the problem, parameters of the optimizer, the dimension $d$, and the total number of iterations $N$. This bound can be made arbitrarily small, and with the right hyper-parameters, Adam can be shown to converge with the same rate of convergence $O(d\ln(N)/\sqrt{N})$. When used with the default parameters, Adam doesn't converge, however, and just like constant step-size SGD, it moves away from the initialization point faster than Adagrad, which might explain its practical success. Finally, we obtain the tightest dependency on the heavy ball momentum decay rate $\beta_1$ among all previous convergence bounds for non-convex Adam and Adagrad, improving from $O((1-\beta_1)^{-3})$ to $O((1-\beta_1)^{-1})$.