Adam Reduces a Unique Form of Sharpness: Theoretical Insights Near the Minimizer Manifold

TL;DR

This paper provides a theoretical analysis showing that Adam implicitly reduces a unique sharpness measure shaped by its adaptive updates, leading to solutions that differ from SGD and can improve sparsity and generalization.

Contribution

It introduces a continuous-time approximation to characterize Adam's behavior, revealing how it minimizes a distinct sharpness measure and extends this analysis to other adaptive optimizers.

Findings

Adam minimizes $ r( ext{Diag}( abla^2 f)^{1/2})$ instead of $ r( abla^2 f)$ as SGD does.

Adam achieves better sparsity and generalization in sparse linear regression with diagonal networks.

The analysis framework applies broadly to various adaptive gradient methods, offering a unified perspective.

Abstract

Despite the popularity of the Adam optimizer in practice, most theoretical analyses study Stochastic Gradient Descent (SGD) as a proxy for Adam, and little is known about how the solutions found by Adam differ. In this paper, we show that Adam implicitly reduces a unique form of sharpness measure shaped by its adaptive updates, leading to qualitatively different solutions from SGD. More specifically, when the training loss is small, Adam wanders around the manifold of minimizers and takes semi-gradients to minimize this sharpness measure in an adaptive manner, a behavior we rigorously characterize through a continuous-time approximation using stochastic differential equations. We further demonstrate how this behavior differs from that of SGD in a well-studied setting: when training overparameterized models with label noise, SGD has been shown to minimize the trace of the Hessian matrix, $\tr(\mH)$, whereas we prove that Adam minimizes $\tr(\Diag(\mH)^{1/2})$ instead. In solving sparse linear regression with diagonal linear networks, this distinction enables Adam to achieve better sparsity and generalization than SGD. Finally, our analysis framework extends beyond Adam to a broad class of adaptive gradient methods, including RMSProp, Adam-mini, Adalayer and Shampoo, and provides a unified perspective on how these adaptive optimizers reduce sharpness, which we hope will offer insights for future optimizer design.