FlowFace++: Explicit Semantic Flow-supervised End-to-End Face Swapping

TL;DR

FlowFace++ introduces an end-to-end face swapping framework that uses explicit semantic flow supervision and a shape-aware discriminator to produce highly realistic swapped faces, especially under challenging conditions.

Contribution

The paper presents a novel face swapping method combining semantic flow supervision, a shape-aware discriminator, and a fine-grained face representation to improve realism and robustness.

Findings

Outperforms state-of-the-art methods in various scenarios.

Effectively handles obstructed or uneven lighting source faces.

Achieves high-quality, shape-aware face swapping results.

Abstract

This work proposes a novel face-swapping framework FlowFace++, utilizing explicit semantic flow supervision and end-to-end architecture to facilitate shape-aware face-swapping. Specifically, our work pretrains a facial shape discriminator to supervise the face swapping network. The discriminator is shape-aware and relies on a semantic flow-guided operation to explicitly calculate the shape discrepancies between the target and source faces, thus optimizing the face swapping network to generate highly realistic results. The face swapping network is a stack of a pre-trained face-masked autoencoder (MAE), a cross-attention fusion module, and a convolutional decoder. The MAE provides a fine-grained facial image representation space, which is unified for the target and source faces and thus facilitates final realistic results. The cross-attention fusion module carries out the source-to-target face swapping in a fine-grained latent space while preserving other attributes of the target image (e.g. expression, head pose, hair, background, illumination, etc). Lastly, the convolutional decoder further synthesizes the swapping results according to the face-swapping latent embedding from the cross-attention fusion module. Extensive quantitative and qualitative experiments on in-the-wild faces demonstrate that our FlowFace++ outperforms the state-of-the-art significantly, particularly while the source face is obstructed by uneven lighting or angle offset.