Associative Memory and Generative Diffusion in the Zero-noise Limit

TL;DR

This paper demonstrates that generative diffusion processes converge to associative memory systems as noise approaches zero, providing a geometric framework to analyze their stability, bifurcations, and robustness across various models.

Contribution

It establishes a universal connection between diffusion processes and associative memory via Morse-Smale dynamical systems, offering a unified geometric perspective on stability and transitions.

Findings

Diffusion processes converge to associative memory models at low noise levels.

Morse-Smale systems are universal approximators of associative memory.

Structural stability ensures robustness of memory and generation landscapes.

Abstract

This paper shows that generative diffusion processes converge to associative memory systems at vanishing noise levels and characterizes the stability, robustness, memorization, and generation dynamics of both model classes. Morse-Smale dynamical systems are shown to be universal approximators of associative memory models, with diffusion processes as their white-noise perturbations. The universal properties of associative memory that follow are used to characterize a generic transition from generation to memory as noise diminishes. Structural stability of Morse-Smale flows -- that is, the robustness of their global critical point structure -- implies the stability of both trajectories and invariant measures for diffusions in the zero-noise limit. The learning and generation landscapes of these models appear as parameterized families of gradient flows and their stochastic perturbations, and the bifurcation theory for Morse-Smale systems implies that they are generically stable except at isolated parameter values, where enumerable sets of local and global bifurcations govern transitions between stable systems in parameter space. These landscapes are thus characterized by ordered bifurcation sequences that create, destroy, or alter connections between rest points and are robust under small stochastic or deterministic perturbations. The framework is agnostic to model formulation, which we verify with examples from energy-based models, denoising diffusion models, and classical and modern Hopfield networks. We additionally derive structural stability criteria for Hopfield-type networks and find that simple cases violate them. Collectively, our geometric approach provides insight into the classification, stability, and emergence of memory and generative landscapes.