Extracting Training Data from Unconditional Diffusion Models

TL;DR

This paper develops a theoretical framework and new methods for extracting training data from diffusion models, revealing their memorization properties and improving data recovery techniques.

Contribution

It introduces a theoretical analysis of memorization in diffusion models and proposes SIDE, a novel data extraction method that outperforms previous approaches.

Findings

SIDE extracts data from unconditional diffusion models where prior methods fail

Theoretical analysis provides new insights into memorization in diffusion models

SIDE achieves over 50% higher effectiveness on CelebA dataset

Abstract

As diffusion probabilistic models (DPMs) are being employed as mainstream models for generative artificial intelligence (AI), the study of their memorization of the raw training data has attracted growing attention. Existing works in this direction aim to establish an understanding of whether or to what extent DPMs learn by memorization. Such an understanding is crucial for identifying potential risks of data leakage and copyright infringement in diffusion models and, more importantly, for more controllable generation and trustworthy application of Artificial Intelligence Generated Content (AIGC). While previous works have made important observations of when DPMs are prone to memorization, these findings are mostly empirical, and the developed data extraction methods only work for conditional diffusion models. In this work, we aim to establish a theoretical understanding of memorization in DPMs with 1) a memorization metric for theoretical analysis, 2) an analysis of conditional memorization with informative and random labels, and 3) two better evaluation metrics for measuring memorization. Based on the theoretical analysis, we further propose a novel data extraction method called \textbf{Surrogate condItional Data Extraction (SIDE)} that leverages a classifier trained on generated data as a surrogate condition to extract training data directly from unconditional diffusion models. Our empirical results demonstrate that SIDE can extract training data from diffusion models where previous methods fail, and it is on average over 50\% more effective across different scales of the CelebA dataset.