3DFlowRenderer: One-shot Face Re-enactment via Dense 3D Facial Flow Estimation

TL;DR

This paper introduces a novel one-shot face re-enactment method using dense 3D facial flow estimation that improves robustness against extreme head poses and enhances background detail reconstruction.

Contribution

It proposes a new warping technology combining 2D and 3D methods with a cyclic warp loss for better geometric control and realistic background reconstruction.

Findings

Outperforms state-of-the-art in artifact-free facial rendering

Robust to extreme head poses

Accurately reconstructs background details

Abstract

Performing facial expression transfer under one-shot setting has been increasing in popularity among research community with a focus on precise control of expressions. Existing techniques showcase compelling results in perceiving expressions, but they lack robustness with extreme head poses. They also struggle to accurately reconstruct background details, thus hindering the realism. In this paper, we propose a novel warping technology which integrates the advantages of both 2D and 3D methods to achieve robust face re-enactment. We generate dense 3D facial flow fields in feature space to warp an input image based on target expressions without depth information. This enables explicit 3D geometric control for re-enacting misaligned source and target faces. We regularize the motion estimation capability of the 3D flow prediction network through proposed "Cyclic warp loss" by converting warped 3D features back into 2D RGB space. To ensure the generation of finer facial region with natural-background, our framework only renders the facial foreground region first and learns to inpaint the blank area which needs to be filled due to source face translation, thus reconstructing the detailed background without any unwanted pixel motion. Extensive evaluation reveals that our method outperforms state-of-the-art techniques in rendering artifact-free facial images.