High-resolution Face Swapping via Latent Semantics Disentanglement

TL;DR

This paper introduces a high-resolution face swapping technique that leverages disentangled latent semantics from a pre-trained GAN to improve quality and consistency in both images and videos.

Contribution

It proposes a novel method to explicitly disentangle semantic attributes in the GAN's latent space for enhanced face swapping results, including extensions to video.

Findings

Outperforms state-of-the-art methods in quality and consistency

Effective disentanglement of structure and appearance attributes

Successful extension to video face swapping with temporal constraints

Abstract

We present a novel high-resolution face swapping method using the inherent prior knowledge of a pre-trained GAN model. Although previous research can leverage generative priors to produce high-resolution results, their quality can suffer from the entangled semantics of the latent space. We explicitly disentangle the latent semantics by utilizing the progressive nature of the generator, deriving structure attributes from the shallow layers and appearance attributes from the deeper ones. Identity and pose information within the structure attributes are further separated by introducing a landmark-driven structure transfer latent direction. The disentangled latent code produces rich generative features that incorporate feature blending to produce a plausible swapping result. We further extend our method to video face swapping by enforcing two spatio-temporal constraints on the latent space and the image space. Extensive experiments demonstrate that the proposed method outperforms state-of-the-art image/video face swapping methods in terms of hallucination quality and consistency. Code can be found at: https://github.com/cnnlstm/FSLSD_HiRes.