Identifying Memorization of Diffusion Models through $p$-Laplace Analysis: Estimators, Bounds and Applications

TL;DR

This paper introduces a novel method using p-Laplace operators derived from diffusion models' score functions to identify memorized training data, providing theoretical bounds and demonstrating effectiveness in image generation tasks.

Contribution

It develops a new approach to detect memorization in diffusion models via p-Laplace analysis, including estimators, bounds, and applications to text-conditioned image generation.

Findings

Effective identification of memorized prompts in image generation

Theoretical error bounds for p-Laplace estimators

Successful application to text-to-image models with unseen conditioning text

Abstract

Diffusion models, today's leading image generative models, estimate the score function, i.e. the gradient of the log probability of (perturbed) data samples, without direct access to the underlying probability distribution. This work investigates whether the estimated score function can be leveraged to compute higher-order differentials, namely the p-Laplace operators. We show that these operators can be employed to identify memorized training data. We propose a numerical p-Laplace approximation based on the learned score functions, showing its effectiveness in identifying key features of the probability landscape. Furthermore, theoretical error-bounds to these estimators are proven and demonstrated numerically. We analyze the structured case of Gaussian mixture models, and demonstrate that the results carry-over to text-conditioned image generative models (text-to-image), where memorization identification based on the p-Laplace operator is performed for the first time, showing its advantage on 500 memorized prompts ($\sim$3000 generated images) in a post-generation regime, especially when the conditioning text is unavailable.