Non-Convergence and Limit Cycles in the Adam optimizer

TL;DR

This paper investigates the convergence properties of the Adam optimizer, revealing that even in simple quadratic cases, limit cycles can occur regardless of hyperparameter choices, challenging previous assumptions about its stability.

Contribution

It extends the convergence analysis of Adam to include bias correction in batch mode, demonstrating the existence of 2-limit-cycles for all hyperparameters in quadratic functions.

Findings

Limit cycles of period 2 exist for Adam with bias correction.

These cycles occur for all hyperparameter choices in quadratic cases.

Stability analysis of these cycles is provided.

Abstract

One of the most popular training algorithms for deep neural networks is the Adaptive Moment Estimation (Adam) introduced by Kingma and Ba. Despite its success in many applications there is no satisfactory convergence analysis: only local convergence can be shown for batch mode under some restrictions on the hyperparameters, counterexamples exist for incremental mode. Recent results show that for simple quadratic objective functions limit cycles of period 2 exist in batch mode, but only for atypical hyperparameters, and only for the algorithm without bias correction. %More general there are several more adaptive gradient methods which try to estimate a fitting learning rate and / or search direction from the training data to improve the learning process compared to pure gradient descent with fixed learningrate. We extend the convergence analysis for Adam in the batch mode with bias correction and show that even for quadratic objective functions as the simplest case of convex functions 2-limit-cycles exist, for all choices of the hyperparameters. We analyze the stability of these limit cycles and relate our analysis to other results where approximate convergence was shown, but under the additional assumption of bounded gradients which does not apply to quadratic functions. The investigation heavily relies on the use of computer algebra due to the complexity of the equations.