ODE approximation for the Adam algorithm: General and overparametrized setting

TL;DR

This paper develops an ODE-based framework to analyze the Adam optimizer, revealing its convergence behavior and ability to find global minima in overparametrized settings, with implications for understanding its empirical success.

Contribution

It introduces an ODE approximation for Adam, providing new theoretical insights into its convergence properties and behavior in overparametrized models.

Findings

Adam is an asymptotic pseudo-trajectory of a specific vector field.

Convergence of Adam implies limits are zeros of the Adam vector field.

In overparametrized settings, Adam can converge to global minima under certain conditions.

Abstract

The Adam optimizer is currently presumably the most popular optimization method in deep learning. In this article we develop an ODE based method to study the Adam optimizer in a fast-slow scaling regime. For fixed momentum parameters and vanishing step-sizes, we show that the Adam algorithm is an asymptotic pseudo-trajectory of the flow of a particular vector field, which is referred to as the Adam vector field. Leveraging properties of asymptotic pseudo-trajectories, we establish convergence results for the Adam algorithm. In particular, in a very general setting we show that if the Adam algorithm converges, then the limit must be a zero of the Adam vector field, rather than a local minimizer or critical point of the objective function. In contrast, in the overparametrized empirical risk minimization setting, the Adam algorithm is able to locally find the set of minima. Specifically, we show that in a neighborhood of the global minima, the objective function serves as a Lyapunov function for the flow induced by the Adam vector field. As a consequence, if the Adam algorithm enters a neighborhood of the global minima infinitely often, it converges to the set of global minima.