Adapt or Forget: Provable Tradeoffs Between Adam and SGD in Nonstationary Optimization

TL;DR

This paper provides a theoretical analysis of Adam's performance in non-stationary stochastic optimization, revealing a noise-drift tradeoff that explains when Adam outperforms or underperforms compared to SGD.

Contribution

It offers the first rigorous bounds characterizing Adam's behavior under non-stationarity, highlighting the impact of hyperparameters and distribution shifts.

Findings

Derived finite-time bounds for Adam's tracking error and stationarity gap.

Identified a noise--drift tradeoff influencing Adam's stability and performance.

Characterized when adaptive step-sizing benefits or harms Adam in non-stationary settings.

Abstract

We provide a theoretical analysis of Adam under non-stationary stochastic objectives, separating two regimes: Euclidean tracking under adaptive strong monotonicity of the Adam-preconditioned mean-gradient operator, and high-probability projected stationarity guarantees under general $L$-smooth objectives. In the tracking regime, we derive finite-time expected and high-probability bounds that decompose sharply into four components: initialization, objective drift, a first-moment tracking error governed by $\beta_1$, and a preconditioner perturbation governed by $\beta_2$. We characterize the burn-in time to reach Adam's irreducible tracking floor under constant and step-decay schedules. We also prove a high-probability bound on the average projected stationarity gap for Adam under distribution shift. Across both analyses, our bounds reveal a noise--drift tradeoff: in noise-dominated regimes, first-moment averaging and adaptive preconditioning can improve the high-probability error, whereas in drift-dominated regimes, stale first-moment information and preconditioner perturbations can compound the cost of nonstationarity, allowing vanilla SGD to achieve a smaller tracking floor. Our explicit $(\beta_1,\beta_2,\epsilon)$-dependent bounds delineate when adaptive step-sizing is beneficial versus harmful, and provide a theoretical mechanism for Adam's empirical instability and stabilization under distribution shift.