Memorization and Regularization in Generative Diffusion Models

TL;DR

This paper analyzes how diffusion models memorize training data and explores regularization techniques to prevent this, providing a theoretical foundation and empirical evaluation of methods like Tikhonov regularization and early stopping.

Contribution

It offers a theoretical analysis of memorization in diffusion models and investigates regularization strategies to mitigate it, advancing understanding of model generalization.

Findings

Regularization can prevent memorization in diffusion models.

Tikhonov regularization promotes better generalization.

Early stopping and under-parameterization reduce data memorization.

Abstract

Diffusion models have emerged as a powerful framework for generative modeling. At the heart of the methodology is score matching: learning gradients of families of log-densities for noisy versions of the data distribution at different scales. When the loss function adopted in score matching is evaluated using empirical data, rather than the population loss, the minimizer corresponds to the score of a time-dependent Gaussian mixture. However, use of this analytically tractable minimizer leads to data memorization: in both unconditioned and conditioned settings, the generative model returns the training samples. This paper contains an analysis of the dynamical mechanism underlying memorization. The analysis highlights the need for regularization to avoid reproducing the analytically tractable minimizer; and, in so doing, lays the foundations for a principled understanding of how to regularize. Numerical experiments investigate the properties of: (i) Tikhonov regularization; (ii) regularization designed to promote asymptotic consistency; and (iii) regularizations induced by under-parameterization of a neural network or by early stopping when training a neural network. These experiments are evaluated in the context of memorization, and directions for future development of regularization are highlighted.