DIAGNOSIS: Detecting Unauthorized Data Usages in Text-to-image Diffusion Models

TL;DR

This paper introduces a novel detection method for identifying unauthorized data usage in text-to-image diffusion models by injecting and detecting unique, nearly imperceptible content in training images.

Contribution

It proposes a stealthy image watermarking technique to detect illegal data usage in diffusion models, validated across multiple training methods and models.

Findings

Effective detection of unauthorized data usage in diffusion models.

Works across various training and fine-tuning methods.

High detection accuracy demonstrated in experiments.

Abstract

Recent text-to-image diffusion models have shown surprising performance in generating high-quality images. However, concerns have arisen regarding the unauthorized data usage during the training or fine-tuning process. One example is when a model trainer collects a set of images created by a particular artist and attempts to train a model capable of generating similar images without obtaining permission and giving credit to the artist. To address this issue, we propose a method for detecting such unauthorized data usage by planting the injected memorization into the text-to-image diffusion models trained on the protected dataset. Specifically, we modify the protected images by adding unique contents on these images using stealthy image warping functions that are nearly imperceptible to humans but can be captured and memorized by diffusion models. By analyzing whether the model has memorized the injected content (i.e., whether the generated images are processed by the injected post-processing function), we can detect models that had illegally utilized the unauthorized data. Experiments on Stable Diffusion and VQ Diffusion with different model training or fine-tuning methods (i.e, LoRA, DreamBooth, and standard training) demonstrate the effectiveness of our proposed method in detecting unauthorized data usages. Code: https://github.com/ZhentingWang/DIAGNOSIS.