Beauty and the Beast: Imperceptible Perturbations Against Diffusion-Based Face Swapping via Directional Attribute Editing

TL;DR

This paper introduces FaceDefense, a novel framework that creates imperceptible adversarial face images to defend against diffusion-based face swapping, balancing protection strength and visual quality.

Contribution

It proposes a new diffusion loss and directional attribute editing to improve defense effectiveness while maintaining imperceptibility, addressing the core trade-off in existing methods.

Findings

FaceDefense outperforms existing defenses in imperceptibility.

It achieves a better balance between protection and image quality.

Extensive experiments validate its superior effectiveness.

Abstract

Diffusion-based face swapping achieves state-of-the-art performance, yet it also exacerbates the potential harm of malicious face swapping to violate portraiture right or undermine personal reputation. This has spurred the development of proactive defense methods. However, existing approaches face a core trade-off: large perturbations distort facial structures, while small ones weaken protection effectiveness. To address these issues, we propose FaceDefense, an enhanced proactive defense framework against diffusion-based face swapping. Our method introduces a new diffusion loss to strengthen the defensive efficacy of adversarial examples, and employs a directional facial attribute editing to restore perturbation-induced distortions, thereby enhancing visual imperceptibility. A two-phase alternating optimization strategy is designed to generate final perturbed face images. Extensive experiments show that FaceDefense significantly outperforms existing methods in both imperceptibility and defense effectiveness, achieving a superior trade-off.