Practical, Generalizable and Robust Backdoor Attacks on Text-to-Image Diffusion Models

TL;DR

This paper introduces a practical, generalizable, and robust backdoor attack on text-to-image diffusion models that requires minimal poisoned data and remains effective against defenses, posing significant security challenges.

Contribution

The authors propose a novel backdoor attack framework that is practical, generalizable across models, and robust against defenses, using only a few poisoned samples to achieve high attack success.

Findings

Achieves over 90% attack success rate with only 10 poisoned samples.

Remains effective against existing backdoor defenses and adaptive strategies.

Maintains high-quality benign image generation despite the attack.

Abstract

Text-to-image diffusion models (T2I DMs) have achieved remarkable success in generating high-quality and diverse images from text prompts, yet recent studies have revealed their vulnerability to backdoor attacks. Existing attack methods suffer from critical limitations: 1) they rely on unnatural adversarial prompts that lack human readability and require massive poisoned data; 2) their effectiveness is typically restricted to specific models, lacking generalizability; and 3) they can be mitigated by recent backdoor defenses. To overcome these challenges, we propose a novel backdoor attack framework that achieves three key properties: 1) \emph{Practicality}: Our attack requires only a few stealthy backdoor samples to generate arbitrary attacker-chosen target images, as well as ensuring high-quality image generation in benign scenarios. 2) \emph{Generalizability:} The attack is applicable across multiple T2I DMs without requiring model-specific redesign. 3) \emph{Robustness:} The attack remains effective against existing backdoor defenses and adaptive defenses. Our extensive experimental results on multiple T2I DMs demonstrate that with only 10 carefully crafted backdoored samples, our attack method achieves $>$90\% attack success rate with negligible degradation in benign image generation quality. We also conduct human evaluation to validate our attack effectiveness. Furthermore, recent backdoor detection and mitigation methods, as well as adaptive defense tailored to our attack are not sufficiently effective, highlighting the pressing need for more robust defense mechanisms against the proposed attack.