Learning Personalized High Quality Volumetric Head Avatars from Monocular RGB Videos

TL;DR

This paper introduces a method to create high-quality, personalized 3D head avatars from monocular RGB videos, combining geometric priors and neural radiance fields for realistic, controllable facial expressions.

Contribution

It presents a hybrid pipeline that integrates 3DMM geometry with neural radiance fields and local feature prediction for detailed, controllable head avatars from monocular videos.

Findings

Achieves high-fidelity, expression-dependent details in avatars.

Demonstrates good generalization to unseen expressions.

Outperforms state-of-the-art methods in rendering quality.

Abstract

We propose a method to learn a high-quality implicit 3D head avatar from a monocular RGB video captured in the wild. The learnt avatar is driven by a parametric face model to achieve user-controlled facial expressions and head poses. Our hybrid pipeline combines the geometry prior and dynamic tracking of a 3DMM with a neural radiance field to achieve fine-grained control and photorealism. To reduce over-smoothing and improve out-of-model expressions synthesis, we propose to predict local features anchored on the 3DMM geometry. These learnt features are driven by 3DMM deformation and interpolated in 3D space to yield the volumetric radiance at a designated query point. We further show that using a Convolutional Neural Network in the UV space is critical in incorporating spatial context and producing representative local features. Extensive experiments show that we are able to reconstruct high-quality avatars, with more accurate expression-dependent details, good generalization to out-of-training expressions, and quantitatively superior renderings compared to other state-of-the-art approaches.