Towards More Realistic Membership Inference Attacks on Large Diffusion Models

TL;DR

This paper investigates the feasibility of membership inference attacks on large diffusion models like Stable Diffusion, highlighting the challenges and proposing a fair evaluation framework to assess attack effectiveness.

Contribution

It introduces a novel, fair evaluation methodology for membership inference attacks on diffusion models and demonstrates that such attacks remain a significant privacy challenge.

Findings

Previous evaluation setups are insufficient for understanding attack effectiveness.

Membership inference attacks on large diffusion models are still largely effective.

Privacy and copyright concerns persist due to the difficulty in defending against these attacks.

Abstract

Generative diffusion models, including Stable Diffusion and Midjourney, can generate visually appealing, diverse, and high-resolution images for various applications. These models are trained on billions of internet-sourced images, raising significant concerns about the potential unauthorized use of copyright-protected images. In this paper, we examine whether it is possible to determine if a specific image was used in the training set, a problem known in the cybersecurity community and referred to as a membership inference attack. Our focus is on Stable Diffusion, and we address the challenge of designing a fair evaluation framework to answer this membership question. We propose a methodology to establish a fair evaluation setup and apply it to Stable Diffusion, enabling potential extensions to other generative models. Utilizing this evaluation setup, we execute membership attacks (both known and newly introduced). Our research reveals that previously proposed evaluation setups do not provide a full understanding of the effectiveness of membership inference attacks. We conclude that the membership inference attack remains a significant challenge for large diffusion models (often deployed as black-box systems), indicating that related privacy and copyright issues will persist in the foreseeable future.