Thinking the Fusion Strategy of Multi-reference Face Reenactment

TL;DR

This paper proposes a multi-reference approach to face reenactment that improves generation quality by using multiple reference images, validated through reconstruction, motion transfer, and new evaluation metrics.

Contribution

Introducing a multi-reference strategy for face reenactment that enhances realism and accuracy over single-reference methods, supported by experiments and a novel evaluation metric.

Findings

Multi-reference approach improves face reconstruction quality.

Enhanced facial motion transfer accuracy.

Proposed evaluation metric validates better performance.

Abstract

In recent advances of deep generative models, face reenactment -manipulating and controlling human face, including their head movement-has drawn much attention for its wide range of applicability. Despite its strong expressiveness, it is inevitable that the models fail to reconstruct or accurately generate unseen side of the face of a given single reference image. Most of existing methods alleviate this problem by learning appearances of human faces from large amount of data and generate realistic texture at inference time. Rather than completely relying on what generative models learn, we show that simple extension by using multiple reference images significantly improves generation quality. We show this by 1) conducting the reconstruction task on publicly available dataset, 2) conducting facial motion transfer on our original dataset which consists of multi-person's head movement video sequences, and 3) using a newly proposed evaluation metric to validate that our method achieves better quantitative results.