Towards Understanding Adam Convergence on Highly Degenerate Polynomials

TL;DR

This paper explores Adam's natural convergence properties on a specific class of highly degenerate polynomials, revealing conditions for automatic convergence and demonstrating its superior local linear convergence compared to other methods.

Contribution

It identifies a class of functions where Adam converges automatically without external schedulers and provides theoretical conditions for its local stability and convergence.

Findings

Adam converges automatically on certain degenerate polynomials.

Adam achieves local linear convergence, outperforming Gradient Descent.

Hyperparameter analysis reveals three behavioral regimes for Adam.

Abstract

Adam is a widely used optimization algorithm in deep learning, yet the specific class of objective functions where it exhibits inherent advantages remains underexplored. Unlike prior studies requiring external schedulers and $\beta_2$ near 1 for convergence, this work investigates the "natural" auto-convergence properties of Adam. We identify a class of highly degenerate polynomials where Adam converges automatically without additional schedulers. Specifically, we derive theoretical conditions for local asymptotic stability on degenerate polynomials and demonstrate strong alignment between theoretical bounds and experimental results. We prove that Adam achieves local linear convergence on these degenerate functions, significantly outperforming the sub-linear convergence of Gradient Descent and Momentum. This acceleration stems from a decoupling mechanism between the second moment $v_t$ and squared gradient $g_t^2$, which exponentially amplifies the effective learning rate. Finally, we characterize Adam's hyperparameter phase diagram, identifying three distinct behavioral regimes: stable convergence, spikes, and SignGD-like oscillation.