BadBlocks: Lightweight and Stealthy Backdoor Threat in Text-to-Image Diffusion Models

TL;DR

BadBlocks introduces a lightweight, stealthy backdoor attack method for text-to-image diffusion models that is highly efficient, effective, and capable of evading current defenses, lowering the barrier for malicious manipulation.

Contribution

It presents BadBlocks, a novel backdoor technique that contaminates specific UNet blocks with minimal computation, enabling stealthy attacks on diffusion models.

Findings

Achieves high attack success with minimal perceptual degradation.

Requires only 30% of the computation and 20% of GPU time of prior methods.

Effectively evades state-of-the-art defenses, especially attention-based detection.

Abstract

Diffusion models have recently achieved remarkable success in image generation, yet growing evidence shows their vulnerability to backdoor attacks, where adversaries implant covert triggers to manipulate outputs. While existing defenses can detect many such attacks via visual inspection and neural network-based analysis, we identify a more lightweight and stealthy threat, termed BadBlocks. BadBlocks selectively contaminates specific blocks within the UNet architecture while preserving the normal behavior of the remaining components. Compared with prior methods, it requires only about 30% of the computation and 20% of the GPU time, yet achieves high attack success rates with minimal perceptual degradation. Extensive experiments demonstrate that BadBlocks can effectively evade state-of-the-art defenses, particularly attention-based detection frameworks. Ablation studies further reveal that effective backdoor injection does not require fine-tuning the entire network and highlight the critical role of certain layers in backdoor mapping. Overall, BadBlocks substantially lowers the barrier for backdooring large-scale diffusion models, even on consumer-grade GPUs.