Invisible Backdoor Attacks on Diffusion Models

TL;DR

This paper introduces a novel optimization framework for creating invisible backdoor triggers in diffusion models, enhancing their stealthiness and applicability in image generation, editing, inpainting, and watermarking.

Contribution

The paper presents the first versatile method for generating invisible backdoor triggers in diffusion models, applicable to both unconditional and conditional settings, including text-guided image tasks.

Findings

The framework successfully creates stealthy backdoors that are hard to detect.

Experiments demonstrate high effectiveness of backdoors across various datasets and samplers.

Backdoors can be used for model watermarking and ownership verification.

Abstract

In recent years, diffusion models have achieved remarkable success in the realm of high-quality image generation, garnering increased attention. This surge in interest is paralleled by a growing concern over the security threats associated with diffusion models, largely attributed to their susceptibility to malicious exploitation. Notably, recent research has brought to light the vulnerability of diffusion models to backdoor attacks, enabling the generation of specific target images through corresponding triggers. However, prevailing backdoor attack methods rely on manually crafted trigger generation functions, often manifesting as discernible patterns incorporated into input noise, thus rendering them susceptible to human detection. In this paper, we present an innovative and versatile optimization framework designed to acquire invisible triggers, enhancing the stealthiness and resilience of inserted backdoors. Our proposed framework is applicable to both unconditional and conditional diffusion models, and notably, we are the pioneers in demonstrating the backdooring of diffusion models within the context of text-guided image editing and inpainting pipelines. Moreover, we also show that the backdoors in the conditional generation can be directly applied to model watermarking for model ownership verification, which further boosts the significance of the proposed framework. Extensive experiments on various commonly used samplers and datasets verify the efficacy and stealthiness of the proposed framework. Our code is publicly available at https://github.com/invisibleTriggerDiffusion/invisible_triggers_for_diffusion.