Disrupting Diffusion: Token-Level Attention Erasure Attack against Diffusion-based Customization

TL;DR

This paper introduces DisDiff, an adversarial attack method that disrupts diffusion-based image customization by erasing cross-attention, effectively preventing malicious misuse while outperforming existing techniques.

Contribution

DisDiff is a novel adversarial attack that targets diffusion models by erasing attention maps and adaptively modulating perturbations, enhancing protection against model misuse.

Findings

DisDiff outperforms state-of-the-art methods by 12.75% in FDFR scores.

DisDiff achieves a 7.25% improvement in ISM scores.

Effective disruption of diffusion model outputs across multiple benchmarks.

Abstract

With the development of diffusion-based customization methods like DreamBooth, individuals now have access to train the models that can generate their personalized images. Despite the convenience, malicious users have misused these techniques to create fake images, thereby triggering a privacy security crisis. In light of this, proactive adversarial attacks are proposed to protect users against customization. The adversarial examples are trained to distort the customization model's outputs and thus block the misuse. In this paper, we propose DisDiff (Disrupting Diffusion), a novel adversarial attack method to disrupt the diffusion model outputs. We first delve into the intrinsic image-text relationships, well-known as cross-attention, and empirically find that the subject-identifier token plays an important role in guiding image generation. Thus, we propose the Cross-Attention Erasure module to explicitly "erase" the indicated attention maps and disrupt the text guidance. Besides,we analyze the influence of the sampling process of the diffusion model on Projected Gradient Descent (PGD) attack and introduce a novel Merit Sampling Scheduler to adaptively modulate the perturbation updating amplitude in a step-aware manner. Our DisDiff outperforms the state-of-the-art methods by 12.75% of FDFR scores and 7.25% of ISM scores across two facial benchmarks and two commonly used prompts on average.