Adynamical systems view of training generativemodels and the memorization phenomenon

TL;DR

This paper offers a system theoretic explanation for the memorization phenomenon in generative models, emphasizing the dynamic aspects of training and two time scale behaviors in stochastic gradient descent.

Contribution

It introduces a dynamical systems perspective to explain memorization in generative models, linking it to time scale separation and loss function dependence.

Findings

Explains memorization as a consequence of two time scales in SGD.

Connects collapse phenomena and double descent to dynamical system properties.

Provides a unified view of phenomena in generative model training.

Abstract

Using recent works of one of the authors (VSB) on collapse in generative models and two time scale dynamics in stochastic gradient descent in high dimensions, we give a system theoretic explanation of the memorization phenomenon in generative models. This relies purely on the dynamic aspects of the training phase. Specifically, we use a result of Austin [2016] to motivate a stylized model for the loss function for stochastic gradient descent (SGD) wherein the loss function has a strong dependence on some variables and weak dependence on the rest in a precise sense. This naturally leads to two distinct time scales in the constant step size SGD that is commonly used in machine learning. This fact has been used to explain the double descent phenomenon in SGD in Borkar [2026]. In conjunction with a mathematical model for collapse phenomenon in SGD developed in Borkar [2025a], we analyze the constant step size SGD using the recent results of Azizian et al. [2024] in order to explain the phenomenon of memorization wherein a generative model that is concurrently being tuned yields the same or similar outputs for significant stretches of time. This gives a novel perspective on the aforementioned phenomena reported in machine learning literature and their interrelationships, using a dynamical systems viewpoint.