Why Adam Can Beat SGD: Second-Moment Normalization Yields Sharper Tails

TL;DR

This paper provides a theoretical explanation for Adam's superior empirical convergence over SGD by analyzing second-moment normalization, showing Adam's better high-probability guarantees.

Contribution

It introduces a novel analysis revealing how second-moment normalization in Adam leads to sharper tail bounds compared to SGD.

Findings

Adam achieves a $oldsymbol{ extstyle rac{1}{ oot 2 ext{ of } oldsymbol{ ext{delta}}}}$ dependence on confidence

First theoretical separation between Adam and SGD in high-probability convergence

Adam's analysis under classical bounded variance model explains empirical performance gap

Abstract

Despite Adam demonstrating faster empirical convergence than SGD in many applications, much of the existing theory yields guarantees essentially comparable to those of SGD, leaving the empirical performance gap insufficiently explained. In this paper, we uncover a key second-moment normalization in Adam and develop a stopping-time/martingale analysis that provably distinguishes Adam from SGD under the classical bounded variance model (a second moment assumption). In particular, we establish the first theoretical separation between the high-probability convergence behaviors of the two methods: Adam achieves a $\delta^{-1/2}$ dependence on the confidence parameter $\delta$, whereas corresponding high-probability guarantee for SGD necessarily incurs at least a $\delta^{-1}$ dependence.