An Inversion-based Measure of Memorization for Diffusion Models

TL;DR

This paper introduces InvMM, an inversion-based measure to accurately quantify memorization in diffusion models, addressing privacy and copyright concerns by providing a reliable auditing tool.

Contribution

The paper presents InvMM, a novel inversion-based metric for measuring memorization in diffusion models, with an adaptive algorithm for accurate estimation across datasets.

Findings

InvMM reliably quantifies memorization in diffusion models.

InvMM reveals the true extent of memorization and its differences from membership.

The method is effective across multiple datasets and model types.

Abstract

The past few years have witnessed substantial advances in image generation powered by diffusion models. However, it was shown that diffusion models are susceptible to training data memorization, raising significant concerns regarding copyright infringement and privacy invasion. This study delves into a rigorous analysis of memorization in diffusion models. We introduce InvMM, an inversion-based measure of memorization, which is based on inverting a sensitive latent noise distribution accounting for the replication of an image. For accurate estimation of the measure, we propose an adaptive algorithm that balances the normality and sensitivity of the noise distribution. Comprehensive experiments across four datasets, conducted on both unconditional and text-guided diffusion models, demonstrate that InvMM provides a reliable and complete quantification of memorization. Notably, InvMM is commensurable between samples, reveals the true extent of memorization from an adversarial standpoint and implies how memorization differs from membership. In practice, it serves as an auditing tool for developers to reliably assess the risk of memorization, thereby contributing to the enhancement of trustworthiness and privacy-preserving capabilities of diffusion models.