RigNeRF: Fully Controllable Neural 3D Portraits

TL;DR

RigNeRF is a novel neural rendering system that enables full control over head pose and facial expressions in 3D portraits, learned from a single short video, surpassing traditional NeRF limitations.

Contribution

We introduce RigNeRF, which integrates a 3D morphable face model with neural radiance fields to allow explicit control of head pose and expressions from minimal input data.

Findings

Effective control of head pose and expressions in 3D portraits.

Achieves photo-realistic novel view synthesis with explicit pose/expression control.

Operates from only a short smartphone video.

Abstract

Volumetric neural rendering methods, such as neural radiance fields (NeRFs), have enabled photo-realistic novel view synthesis. However, in their standard form, NeRFs do not support the editing of objects, such as a human head, within a scene. In this work, we propose RigNeRF, a system that goes beyond just novel view synthesis and enables full control of head pose and facial expressions learned from a single portrait video. We model changes in head pose and facial expressions using a deformation field that is guided by a 3D morphable face model (3DMM). The 3DMM effectively acts as a prior for RigNeRF that learns to predict only residuals to the 3DMM deformations and allows us to render novel (rigid) poses and (non-rigid) expressions that were not present in the input sequence. Using only a smartphone-captured short video of a subject for training, we demonstrate the effectiveness of our method on free view synthesis of a portrait scene with explicit head pose and expression controls. The project page can be found here: http://shahrukhathar.github.io/2022/06/06/RigNeRF.html