Rigorous Asymptotics for First-Order Algorithms Through the Dynamical Cavity Method

TL;DR

This paper rigorously formalizes the dynamical cavity method to derive asymptotic equations for the dynamics of broad classes of first-order algorithms, bridging physics-inspired methods with mathematical proofs.

Contribution

It provides the first rigorous mathematical foundation for the dynamical cavity method applied to General First Order Methods, including Gradient Descent and Approximate Message Passing.

Findings

Rigorous proof of DMFT equations for first-order algorithms.

Formalization of the dynamical cavity method.

Applicability to a broad class of high-dimensional algorithms.

Abstract

Dynamical Mean Field Theory (DMFT) provides an asymptotic description of the dynamics of macroscopic observables in certain disordered systems. Originally pioneered in the context of spin glasses by Sompolinsky and Zippelius (1982), it has since been used to derive asymptotic dynamical equations for a wide range of models in physics, high-dimensional statistics and machine learning. One of the main tools used by physicists to obtain these equations is the dynamical cavity method, which has remained largely non-rigorous. In contrast, existing mathematical formalizations have relied on alternative approaches, including Gaussian conditioning, large deviations over paths, or Fourier analysis. In this work, we formalize the dynamical cavity method and use it to give a new proof of the DMFT equations for General First Order Methods, a broad class of dynamics encompassing algorithms such as Gradient Descent and Approximate Message Passing.