On the Edge of Memorization in Diffusion Models

TL;DR

This paper investigates when diffusion models memorize training data versus generalize beyond it, providing a theoretical framework and experimental validation for understanding the phase transition between these behaviors based on model size.

Contribution

It introduces a mathematical laboratory for analyzing memorization and generalization in diffusion models, and predicts the critical model size where memorization dominates.

Findings

Theoretical characterization of the crossover point for memorization versus generalization.

Experimental validation of the phase transition in diffusion models.

Analytical prediction of the model size threshold for memorization.

Abstract

When do diffusion models reproduce their training data, and when are they able to generate samples beyond it? A practically relevant theoretical understanding of this interplay between memorization and generalization may significantly impact real-world deployments of diffusion models with respect to issues such as copyright infringement and data privacy. In this work, to disentangle the different factors that influence memorization and generalization in practical diffusion models, we introduce a scientific and mathematical "laboratory" for investigating these phenomena in diffusion models trained on fully synthetic or natural image-like structured data. Within this setting, we hypothesize that the memorization or generalization behavior of an underparameterized trained model is determined by the difference in training loss between an associated memorizing model and a generalizing model. To probe this hypothesis, we theoretically characterize a crossover point wherein the weighted training loss of a fully generalizing model becomes greater than that of an underparameterized memorizing model at a critical value of model (under)parameterization. We then demonstrate via carefully-designed experiments that the location of this crossover predicts a phase transition in diffusion models trained via gradient descent, validating our hypothesis. Ultimately, our theory enables us to analytically predict the model size at which memorization becomes predominant. Our work provides an analytically tractable and practically meaningful setting for future theoretical and empirical investigations. Code for our experiments is available at https://github.com/DruvPai/diffusion_mem_gen.