From Adam to Adam-Like Lagrangians: Second-Order Nonlocal Dynamics

TL;DR

This paper develops a novel second-order nonlocal dynamical system model for Adam optimization, providing theoretical analysis and a variational perspective, with numerical validation on Rosenbrock functions.

Contribution

It introduces a second-order integro-differential formulation of Adam and a nonlocal Lagrangian perspective, extending the understanding of Adam's dynamics.

Findings

The inertial nonlocal model relates to the first-order Adam flow via an $\alpha$-refinement limit.

Lyapunov-based stability and convergence are established for the proposed dynamics.

Numerical simulations confirm the model's agreement with discrete Adam on Rosenbrock-type problems.

Abstract

In this paper, we derive an accelerated continuous-time formulation of Adam by modeling it as a second-order integro-differential dynamical system. We relate this inertial nonlocal model to an existing first-order nonlocal Adam flow through an $\alpha$-refinement limit, and we provide Lyapunov-based stability and convergence analyses. We also introduce an Adam-inspired nonlocal Lagrangian formulation, offering a variational viewpoint. Numerical simulations on Rosenbrock-type examples show agreement between the proposed dynamics and discrete Adam.