Privacy Protection in Personalized Diffusion Models via Targeted Cross-Attention Adversarial Attack

TL;DR

This paper introduces CoPSAM, an adversarial attack method that selectively manipulates cross-attention layers in personalized diffusion models to protect user privacy by preventing unauthorized content use.

Contribution

The paper presents a novel, efficient adversarial attack targeting cross-attention layers in T2I diffusion models for privacy protection, outperforming existing methods.

Findings

Outperforms existing privacy protection methods.

Achieves better protection with lower noise levels.

Effectively prevents unauthorized content use.

Abstract

The growing demand for customized visual content has led to the rise of personalized text-to-image (T2I) diffusion models. Despite their remarkable potential, they pose significant privacy risk when misused for malicious purposes. In this paper, we propose a novel and efficient adversarial attack method, Concept Protection by Selective Attention Manipulation (CoPSAM) which targets only the cross-attention layers of a T2I diffusion model. For this purpose, we carefully construct an imperceptible noise to be added to clean samples to get their adversarial counterparts. This is obtained during the fine-tuning process by maximizing the discrepancy between the corresponding cross-attention maps of the user-specific token and the class-specific token, respectively. Experimental validation on a subset of CelebA-HQ face images dataset demonstrates that our approach outperforms existing methods. Besides this, our method presents two important advantages derived from the qualitative evaluation: (i) we obtain better protection results for lower noise levels than our competitors; and (ii) we protect the content from unauthorized use thereby protecting the individual's identity from potential misuse.