Absorbing state dynamics of stochastic gradient descent

TL;DR

This paper models stochastic gradient descent (SGD) as a biased random organization process, revealing an absorbing phase transition and universality class behavior that elucidates neural manifold packing dynamics in deep learning.

Contribution

It introduces a minimal physical model of SGD dynamics using particle packing and connects it to nonequilibrium absorbing state models, providing new insights into neural manifold separation.

Findings

SGD undergoes an absorbing phase transition similar to BRO models.

Near the critical point, SGD exhibits Manna universality class behavior.

SGD dynamics converge to a critical packing fraction of approximately 0.64.

Abstract

Stochastic gradient descent (SGD) is a fundamental tool for training deep neural networks across a variety of tasks. In self-supervised learning, different input categories map to distinct manifolds in the embedded neural state space. Accurate classification is achieved by separating these manifolds during learning, akin to a packing problem. We investigate the dynamics of ``neural manifold packing'' by employing a minimal model in which SGD is applied to spherical particles in physical space. In this model, SGD minimizes the system's energy (classification loss) by stochastically reducing overlaps between particles (manifolds). We observe that this process undergoes an absorbing phase transition, prompting us to use the framework of biased random organization (BRO), a nonequilibrium absorbing state model, to describe SGD behavior. We show that BRO dynamics can be approximated by those of particles with linear repulsive interactions under multiplicative anisotropic noise. Thus, for a linear repulsive potential and small kick sizes (learning rates), we find that BRO and SGD become equivalent, converging to the same critical packing fraction $\phi_c \approx 0.64$, despite the fundamentally different origins of their noise. This equivalence is further supported by the observation that, like BRO, near the critical point, SGD exhibits behavior consistent with the Manna universality class. Above the critical point, SGD exhibits a bias towards flatter minima of the energy landscape, reinforcing the analogy with loss minimization in neural networks.