Uniform a priori bounds and error analysis for the Adam stochastic gradient descent optimization method

TL;DR

This paper establishes uniform bounds and provides the first unconditional error analysis for the Adam optimizer in strongly convex stochastic optimization problems, enhancing theoretical understanding of its convergence behavior.

Contribution

It introduces uniform a priori bounds for Adam, enabling an unconditional error analysis for a broad class of strongly convex stochastic optimization problems.

Findings

First unconditional error analysis for Adam in strongly convex SOPs.

Established uniform a priori bounds for Adam.

Enhanced theoretical understanding of Adam's convergence.

Abstract

The adaptive moment estimation (Adam) optimizer proposed by Kingma & Ba (2014) is presumably the most popular stochastic gradient descent (SGD) optimization method for the training of deep neural networks (DNNs) in artificial intelligence (AI) systems. Despite its groundbreaking success in the training of AI systems, it still remains an open research problem to provide a complete error analysis of Adam, not only for optimizing DNNs but even when applied to strongly convex stochastic optimization problems (SOPs). Previous error analysis results for strongly convex SOPs in the literature provide conditional convergence analyses that rely on the assumption that Adam does not diverge to infinity but remains uniformly bounded. It is the key contribution of this work to establish uniform a priori bounds for Adam and, thereby, to provide -- for the first time -- an unconditional error analysis for Adam for a large class of strongly convex SOPs.