Dual Attention Guided Defense Against Malicious Edits

TL;DR

This paper introduces DANP, a dual attention-guided noise perturbation method that enhances the robustness of text-to-image diffusion models against malicious editing by disrupting their semantic understanding and generation processes.

Contribution

The paper presents a novel dual attention-guided noise perturbation technique that effectively defends against malicious edits in diffusion-based image editing models.

Findings

DANP significantly reduces the success rate of malicious edits.

The method achieves state-of-the-art immunity performance.

Extensive experiments validate its effectiveness across various scenarios.

Abstract

Recent progress in text-to-image diffusion models has transformed image editing via text prompts, yet this also introduces significant ethical challenges from potential misuse in creating deceptive or harmful content. While current defenses seek to mitigate this risk by embedding imperceptible perturbations, their effectiveness is limited against malicious tampering. To address this issue, we propose a Dual Attention-Guided Noise Perturbation (DANP) immunization method that adds imperceptible perturbations to disrupt the model's semantic understanding and generation process. DANP functions over multiple timesteps to manipulate both cross-attention maps and the noise prediction process, using a dynamic threshold to generate masks that identify text-relevant and irrelevant regions. It then reduces attention in relevant areas while increasing it in irrelevant ones, thereby misguides the edit towards incorrect regions and preserves the intended targets. Additionally, our method maximizes the discrepancy between the injected noise and the model's predicted noise to further interfere with the generation. By targeting both attention and noise prediction mechanisms, DANP exhibits impressive immunity against malicious edits, and extensive experiments confirm that our method achieves state-of-the-art performance.