BackdoorDM: A Comprehensive Benchmark for Backdoor Learning on Diffusion Model

TL;DR

BackdoorDM is the first comprehensive benchmark for evaluating backdoor learning methods on diffusion models, including attack and defense strategies, visualization tools, and evaluation metrics, to facilitate fair comparison and future research.

Contribution

It introduces a unified framework and benchmark for backdoor learning on diffusion models, covering attack types, defense strategies, and evaluation methods, which were lacking in prior work.

Findings

Systematic classification of backdoor attack and target types.

Evaluation of nine attack and four defense methods.

Insights into the effectiveness of different backdoor strategies.

Abstract

Backdoor learning is a critical research topic for understanding the vulnerabilities of deep neural networks. While the diffusion model (DM) has been broadly deployed in public over the past few years, the understanding of its backdoor vulnerability is still in its infancy compared to the extensive studies in discriminative models. Recently, many different backdoor attack and defense methods have been proposed for DMs, but a comprehensive benchmark for backdoor learning on DMs is still lacking. This absence makes it difficult to conduct fair comparisons and thorough evaluations of the existing approaches, thus hindering future research progress. To address this issue, we propose \textit{BackdoorDM}, the first comprehensive benchmark designed for backdoor learning on DMs. It comprises nine state-of-the-art (SOTA) attack methods, four SOTA defense strategies, and three useful visualization analysis tools. We first systematically classify and formulate the existing literature in a unified framework, focusing on three different backdoor attack types and five backdoor target types, which are restricted to a single type in discriminative models. Then, we systematically summarize the evaluation metrics for each type and propose a unified backdoor evaluation method based on multimodal large language model (MLLM). Finally, we conduct a comprehensive evaluation and highlight several important conclusions. We believe that BackdoorDM will help overcome current barriers and contribute to building a trustworthy artificial intelligence generated content (AIGC) community. The codes are released in https://github.com/linweiii/BackdoorDM.