Memorization to Generalization: Emergence of Diffusion Models from Associative Memory

TL;DR

This paper explores how diffusion models transition from memorizing training data to generalizing, using Dense Associative Memories theory to identify emergent spurious states as signs of generative capabilities.

Contribution

It provides a novel perspective by analyzing diffusion models as associative memories, revealing the emergence of spurious states during the memorization-to-generalization transition.

Findings

Diffusion models create distinct attractors for training samples in small data regimes.

Spurious states emerge as the data size increases, indicating a shift towards generalization.

Spurious states are associated with the first signs of generative capabilities in diffusion models.

Abstract

Dense Associative Memories (DenseAMs) are generalizations of Hopfield networks, which have superior information storage capacity and can store training data points (memories) at local minima of the energy landscape. When the amount of training data exceeds the critical memory storage capacity of these models, new local minima, which are different from the training data, emerge. In Associative Memory these emergent local minima are called $\textit{spurious}\; \textit{states}$, which hinder memory retrieval. In this work, we examine diffusion models (DMs) through the DenseAM lens, viewing their generative process as an attempt of a memory retrieval. In the small data regimes, DMs create distinct attractors for each training sample, akin to DenseAMs below the critical memory storage. As the training data size increases, they transition from memorization to generalization. We identify a critical intermediate phase, predicted by DenseAM theory -- the spurious states. In generative modeling, these states are no longer negative artifacts but rather are the first signs of generative capabilities. We characterize the basins of attraction, energy landscape curvature, and computational properties of these previously overlooked states. Their existence is demonstrated across a wide range of architectures and datasets.