FLAME-in-NeRF : Neural control of Radiance Fields for Free View Face Animation

TL;DR

This paper introduces FLAME-in-NeRF, a neural rendering approach that enables free-view portrait video synthesis with explicit facial expression control, using a low-dimensional expression space and minimal training data.

Contribution

It combines neural radiance fields with 3D morphable face models to achieve controllable, photorealistic portrait video synthesis from short videos.

Findings

Effective free-view synthesis of portrait videos.

Explicit control of facial expressions through 3DMM.

Requires only short video for training.

Abstract

This paper presents a neural rendering method for controllable portrait video synthesis. Recent advances in volumetric neural rendering, such as neural radiance fields (NeRF), has enabled the photorealistic novel view synthesis of static scenes with impressive results. However, modeling dynamic and controllable objects as part of a scene with such scene representations is still challenging. In this work, we design a system that enables both novel view synthesis for portrait video, including the human subject and the scene background, and explicit control of the facial expressions through a low-dimensional expression representation. We leverage the expression space of a 3D morphable face model (3DMM) to represent the distribution of human facial expressions, and use it to condition the NeRF volumetric function. Furthermore, we impose a spatial prior brought by 3DMM fitting to guide the network to learn disentangled control for scene appearance and facial actions. We demonstrate the effectiveness of our method on free view synthesis of portrait videos with expression controls. To train a scene, our method only requires a short video of a subject captured by a mobile device.