UAdam: Unified Adam-Type Algorithmic Framework for Non-Convex Stochastic Optimization

TL;DR

This paper introduces UAdam, a unified framework for Adam-type algorithms, providing convergence guarantees in non-convex stochastic optimization and encompassing many variants like Adam, NAdam, and AMSGrad.

Contribution

The paper presents a general framework for Adam-type algorithms with convergence analysis, unifying various variants and establishing theoretical guarantees in non-convex stochastic settings.

Findings

UAdam converges to stationary point neighborhoods at rate O(1/T).

The neighborhood size decreases as the momentum parameter β increases.

Vanilla Adam can converge with proper hyperparameters, according to the analysis.

Abstract

Adam-type algorithms have become a preferred choice for optimisation in the deep learning setting, however, despite success, their convergence is still not well understood. To this end, we introduce a unified framework for Adam-type algorithms (called UAdam). This is equipped with a general form of the second-order moment, which makes it possible to include Adam and its variants as special cases, such as NAdam, AMSGrad, AdaBound, AdaFom, and Adan. This is supported by a rigorous convergence analysis of UAdam in the non-convex stochastic setting, showing that UAdam converges to the neighborhood of stationary points with the rate of $\mathcal{O}(1/T)$. Furthermore, the size of neighborhood decreases as $\beta$ increases. Importantly, our analysis only requires the first-order momentum factor to be close enough to 1, without any restrictions on the second-order momentum factor. Theoretical results also show that vanilla Adam can converge by selecting appropriate hyperparameters, which provides a theoretical guarantee for the analysis, applications, and further developments of the whole class of Adam-type algorithms.