Automatic Face Reenactment

TL;DR

This paper introduces an automatic, image-based facial reenactment system that replaces an actor's face in a video with a user's face from a short source video, maintaining the original performance without needing a 3D model.

Contribution

It presents a novel reenactment pipeline combining image retrieval with appearance and motion matching, enabling convincing results from short, in-the-wild source videos without complex modeling.

Findings

System produces realistic reenactments from short source videos.

It is robust to head motion and does not require source and target to be similar.

Effective on low-quality and internet-sourced footage.

Abstract

We propose an image-based, facial reenactment system that replaces the face of an actor in an existing target video with the face of a user from a source video, while preserving the original target performance. Our system is fully automatic and does not require a database of source expressions. Instead, it is able to produce convincing reenactment results from a short source video captured with an off-the-shelf camera, such as a webcam, where the user performs arbitrary facial gestures. Our reenactment pipeline is conceived as part image retrieval and part face transfer: The image retrieval is based on temporal clustering of target frames and a novel image matching metric that combines appearance and motion to select candidate frames from the source video, while the face transfer uses a 2D warping strategy that preserves the user's identity. Our system excels in simplicity as it does not rely on a 3D face model, it is robust under head motion and does not require the source and target performance to be similar. We show convincing reenactment results for videos that we recorded ourselves and for low-quality footage taken from the Internet.