Fokker-Planck Analysis and Invariant Laws for a Continuous-Time Stochastic Model of Adam-Type Dynamics

TL;DR

This paper models Adam-type optimization algorithms using continuous-time stochastic differential equations, analyzes their invariant measures, and proves exponential convergence under certain conditions.

Contribution

It introduces a continuous-time stochastic model for Adam-type methods, analyzes the associated Fokker-Planck equation, and establishes convergence properties.

Findings

Existence and uniqueness of invariant measures under mild conditions.

Explicit characterization of noise propagation via matrix A(x).

Proof of exponential convergence to invariant measure.

Abstract

We develop a continuous-time model for the long-term dynamics of adaptive stochastic optimization, focusing on bias-corrected Adam-type methods. Starting from a finite-sum setting, we identify a canonical scaling of learning rates, decay parameters, and gradient noise that yields a coupled, time-inhomogeneous stochastic differential equation for the parameters $x_t$, first-moment tracker $z_t$, and second-moment tracker $y_t$. Bias correction persists via explicit time-dependent coefficients, and the dynamics becomes asymptotically time-homogeneous. We analyze the associated Fokker-Planck equation and, under mild regularity and dissipativity assumptions on $f$, prove existence and uniqueness of invariant measures. Noise propagation is governed by $A(x)=\mathrm{Diag}(\nabla f(x))H_f(x)$. Hypoellipticity may fail on $\mathcal D_A\times\mathbb R^m\times(\mathbb R_+)^m$, where \[ \mathcal D_A=\{x\in\mathbb R^m:\exists j,\ e_j^\top A(x)=0\}\subset\{x:\det A(x)=0\}=\mathcal D_A^\dagger, \] and critical points of $f$ lie in $\mathcal D_A$. We show $\mathcal D_A^\dagger\neq\mathbb R^m$ and use this to prove exponential convergence of the Markov semigroup $\mu_0P_t$ to a unique invariant measure, uniformly in $\mu_0$. The proof uses a Harris-type argument, minorization on Lyapunov sublevel sets, control constructions, and hypoellipticity on $(\mathbb R^m\setminus\mathcal D_A)\times\mathbb R^m\times(\mathbb R_+)^m$. This provides a transparent continuous-time view of Adam-type dynamics.