High-fidelity Facial Avatar Reconstruction from Monocular Video with Generative Priors

TL;DR

This paper introduces a novel NeRF-based facial avatar reconstruction method using personalized 3D-aware generative priors, enabling high-fidelity, photo-realistic rendering and face reenactment from monocular videos with improved results.

Contribution

It proposes a new approach to learn personalized generative priors in a low-dimensional latent space for facial avatars, enhancing reconstruction and reenactment fidelity.

Findings

Superior novel view synthesis results

Faithful face reenactment performance

Applicable to various input signals

Abstract

High-fidelity facial avatar reconstruction from a monocular video is a significant research problem in computer graphics and computer vision. Recently, Neural Radiance Field (NeRF) has shown impressive novel view rendering results and has been considered for facial avatar reconstruction. However, the complex facial dynamics and missing 3D information in monocular videos raise significant challenges for faithful facial reconstruction. In this work, we propose a new method for NeRF-based facial avatar reconstruction that utilizes 3D-aware generative prior. Different from existing works that depend on a conditional deformation field for dynamic modeling, we propose to learn a personalized generative prior, which is formulated as a local and low dimensional subspace in the latent space of 3D-GAN. We propose an efficient method to construct the personalized generative prior based on a small set of facial images of a given individual. After learning, it allows for photo-realistic rendering with novel views and the face reenactment can be realized by performing navigation in the latent space. Our proposed method is applicable for different driven signals, including RGB images, 3DMM coefficients, and audios. Compared with existing works, we obtain superior novel view synthesis results and faithfully face reenactment performance.