LDFaceNet: Latent Diffusion-based Network for High-Fidelity Deepfake Generation

TL;DR

LDFaceNet introduces a novel latent diffusion-based face swapping method that produces highly realistic and diverse deepfake images without requiring retraining, leveraging facial segmentation and recognition for guided denoising.

Contribution

This paper is the first to apply latent diffusion models to face swapping, offering a new approach that enhances realism and diversity without retraining.

Findings

Produces highly realistic face swaps

Offers greater diversity in generated images

Operates without retraining the model

Abstract

Over the past decade, there has been tremendous progress in the domain of synthetic media generation. This is mainly due to the powerful methods based on generative adversarial networks (GANs). Very recently, diffusion probabilistic models, which are inspired by non-equilibrium thermodynamics, have taken the spotlight. In the realm of image generation, diffusion models (DMs) have exhibited remarkable proficiency in producing both realistic and heterogeneous imagery through their stochastic sampling procedure. This paper proposes a novel facial swapping module, termed as LDFaceNet (Latent Diffusion based Face Swapping Network), which is based on a guided latent diffusion model that utilizes facial segmentation and facial recognition modules for a conditioned denoising process. The model employs a unique loss function to offer directional guidance to the diffusion process. Notably, LDFaceNet can incorporate supplementary facial guidance for desired outcomes without any retraining. To the best of our knowledge, this represents the first application of the latent diffusion model in the face-swapping task without prior training. The results of this study demonstrate that the proposed method can generate extremely realistic and coherent images by leveraging the potential of the diffusion model for facial swapping, thereby yielding superior visual outcomes and greater diversity.