On the Convergence of Adam and Beyond
Sashank J. Reddi, Satyen Kale, Sanjiv Kumar

TL;DR
This paper analyzes why Adam and similar algorithms sometimes fail to converge in deep learning, identifies the cause as the exponential moving average, and proposes new variants with better convergence and performance.
Contribution
It provides a clear example of Adam's convergence failure and introduces modified algorithms with long-term memory to ensure convergence and improve empirical results.
Findings
Adam can fail to converge even in simple convex settings
The exponential moving average causes convergence issues
New Adam variants with long-term memory improve convergence and performance
Abstract
Several recently proposed stochastic optimization methods that have been successfully used in training deep networks such as RMSProp, Adam, Adadelta, Nadam are based on using gradient updates scaled by square roots of exponential moving averages of squared past gradients. In many applications, e.g. learning with large output spaces, it has been empirically observed that these algorithms fail to converge to an optimal solution (or a critical point in nonconvex settings). We show that one cause for such failures is the exponential moving average used in the algorithms. We provide an explicit example of a simple convex optimization setting where Adam does not converge to the optimal solution, and describe the precise problems with the previous analysis of Adam algorithm. Our analysis suggests that the convergence issues can be fixed by endowing such algorithms with `long-term memory' of…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsTheology and Philosophy of Evil
MethodsNADAM · AdaDelta · RMSProp · AMSGrad · Adam
