Facial Reenactment Through a Personalized Generator

TL;DR

This paper introduces a personalized generative model for facial reenactment that uses a short video of an individual to produce highly faithful, identity-preserving images, enabling accurate pose and expression transfer with semantic editing capabilities.

Contribution

The paper presents a novel personalized generator trained on a short video, reducing reenactment to pose and expression mimicry, with state-of-the-art results and semantic editing in the latent space.

Findings

Achieves state-of-the-art facial reenactment performance.

Ensures identity preservation in generated images.

Enables semantic editing and stylization of reenacted faces.

Abstract

In recent years, the role of image generative models in facial reenactment has been steadily increasing. Such models are usually subject-agnostic and trained on domain-wide datasets. The appearance of the reenacted individual is learned from a single image, and hence, the entire breadth of the individual's appearance is not entirely captured, leading these methods to resort to unfaithful hallucination. Thanks to recent advancements, it is now possible to train a personalized generative model tailored specifically to a given individual. In this paper, we propose a novel method for facial reenactment using a personalized generator. We train the generator using frames from a short, yet varied, self-scan video captured using a simple commodity camera. Images synthesized by the personalized generator are guaranteed to preserve identity. The premise of our work is that the task of reenactment is thus reduced to accurately mimicking head poses and expressions. To this end, we locate the desired frames in the latent space of the personalized generator using carefully designed latent optimization. Through extensive evaluation, we demonstrate state-of-the-art performance for facial reenactment. Furthermore, we show that since our reenactment takes place in a semantic latent space, it can be semantically edited and stylized in post-processing.