Safeguarding Facial Identity against Diffusion-based Face Swapping via Cascading Pathway Disruption

TL;DR

VoidFace is a systemic defense method that disrupts diffusion-based face swapping pipelines by injecting targeted perturbations, effectively safeguarding facial identity while maintaining high visual quality.

Contribution

We introduce VoidFace, a novel cascading disruption approach that targets multiple stages of diffusion-based face swapping to enhance privacy protection.

Findings

VoidFace outperforms existing defenses in effectiveness.

It maintains high visual quality of adversarial faces.

It generalizes across various diffusion models.

Abstract

The rapid evolution of diffusion models has democratized face swapping but also raises concerns about privacy and identity security. Existing proactive defenses, often adapted from image editing attacks, prove ineffective in this context. We attribute this failure to an oversight of the structural resilience and the unique static conditional guidance mechanism inherent in face swapping systems. To address this, we propose VoidFace, a systemic defense method that views face swapping as a coupled identity pathway. By injecting perturbations at critical bottlenecks, VoidFace induces cascading disruption throughout the pipeline. Specifically, we first introduce localization disruption and identity erasure to degrade physical regression and semantic embeddings, thereby impairing the accurate modeling of the source face. We then intervene in the generative domain by decoupling attention mechanisms to sever identity injection, and corrupting intermediate diffusion features to prevent the reconstruction of source identity. To ensure visual imperceptibility, we perform adversarial search in the latent manifold, guided by a perceptual adaptive strategy to balance attack potency with image quality. Extensive experiments show that VoidFace outperforms existing defenses across various diffusion-based swapping models, while producing adversarial faces with superior visual quality.