Real-World Benchmarks Make Membership Inference Attacks Fail on Diffusion Models

TL;DR

This paper critically evaluates membership inference attacks on diffusion models, revealing their limitations in real-world scenarios and introducing a new benchmark, CopyMark, for more realistic assessment.

Contribution

It uncovers overestimated MIA performance on diffusion models and presents CopyMark, a benchmark for fairer, more practical evaluation of MIAs.

Findings

Current MIAs perform poorly under realistic conditions.

Existing evaluations overstate MIA effectiveness.

MIA reliability is limited for protecting diffusion model data.

Abstract

Membership inference attacks (MIAs) on diffusion models have emerged as potential evidence of unauthorized data usage in training pre-trained diffusion models. These attacks aim to detect the presence of specific images in training datasets of diffusion models. Our study delves into the evaluation of state-of-the-art MIAs on diffusion models and reveals critical flaws and overly optimistic performance estimates in existing MIA evaluation. We introduce CopyMark, a more realistic MIA benchmark that distinguishes itself through the support for pre-trained diffusion models, unbiased datasets, and fair evaluation pipelines. Through extensive experiments, we demonstrate that the effectiveness of current MIA methods significantly degrades under these more practical conditions. Based on our results, we alert that MIA, in its current state, is not a reliable approach for identifying unauthorized data usage in pre-trained diffusion models. To the best of our knowledge, we are the first to discover the performance overestimation of MIAs on diffusion models and present a unified benchmark for more realistic evaluation. Our code is available on GitHub: \url{https://github.com/caradryanl/CopyMark}.