DiffFace: Diffusion-based Face Swapping with Facial Guidance

TL;DR

DiffFace introduces a novel diffusion-based framework for face swapping that leverages facial guidance and target-preserving blending, achieving high fidelity, diversity, and controllability, surpassing previous GAN-based methods.

Contribution

This is the first application of diffusion models to face swapping, offering improved stability, fidelity, and flexibility without re-training.

Findings

Achieves superior or comparable results to state-of-the-art methods.

Demonstrates high fidelity and diversity in face swapping.

Provides flexible control over identity attributes.

Abstract

In this paper, we propose a diffusion-based face swapping framework for the first time, called DiffFace, composed of training ID conditional DDPM, sampling with facial guidance, and a target-preserving blending. In specific, in the training process, the ID conditional DDPM is trained to generate face images with the desired identity. In the sampling process, we use the off-the-shelf facial expert models to make the model transfer source identity while preserving target attributes faithfully. During this process, to preserve the background of the target image and obtain the desired face swapping result, we additionally propose a target-preserving blending strategy. It helps our model to keep the attributes of the target face from noise while transferring the source facial identity. In addition, without any re-training, our model can flexibly apply additional facial guidance and adaptively control the ID-attributes trade-off to achieve the desired results. To the best of our knowledge, this is the first approach that applies the diffusion model in face swapping task. Compared with previous GAN-based approaches, by taking advantage of the diffusion model for the face swapping task, DiffFace achieves better benefits such as training stability, high fidelity, diversity of the samples, and controllability. Extensive experiments show that our DiffFace is comparable or superior to the state-of-the-art methods on several standard face swapping benchmarks.