Auditing Data Provenance in Real-world Text-to-Image Diffusion Models for Privacy and Copyright Protection

TL;DR

This paper introduces a black-box auditing framework called FSCA for verifying data provenance in text-to-image diffusion models, enhancing privacy and copyright protection without needing internal model access.

Contribution

The paper presents a novel black-box auditing method that leverages semantic connections within diffusion models, outperforming existing approaches in real-world scenarios.

Findings

FSCA surpasses state-of-the-art baselines across various metrics.

Achieves up to 90% user-level accuracy with only 10 samples per user.

Effective in real-world datasets like LAION-mi and COCO.

Abstract

Text-to-image diffusion model since its propose has significantly influenced the content creation due to its impressive generation capability. However, this capability depends on large-scale text-image datasets gathered from web platforms like social media, posing substantial challenges in copyright compliance and personal privacy leakage. Though there are some efforts devoted to explore approaches for auditing data provenance in text-to-image diffusion models, existing work has unrealistic assumptions that can obtain model internal knowledge, e.g., intermediate results, or the evaluation is not reliable. To fill this gap, we propose a completely black-box auditing framework called Feature Semantic Consistency-based Auditing (FSCA). It utilizes two types of semantic connections within the text-to-image diffusion model for auditing, eliminating the need for access to internal knowledge. To demonstrate the effectiveness of our FSCA framework, we perform extensive experiments on LAION-mi dataset and COCO dataset, and compare with eight state-of-the-art baseline approaches. The results show that FSCA surpasses previous baseline approaches across various metrics and different data distributions, showcasing the superiority of our FSCA. Moreover, we introduce a recall balance strategy and a threshold adjustment strategy, which collectively allows FSCA to reach up a user-level accuracy of 90% in a real-world auditing scenario with only 10 samples/user, highlighting its strong auditing potential in real-world applications. Our code is made available at https://github.com/JiePKU/FSCA.