3DMM-RF: Convolutional Radiance Fields for 3D Face Modeling

TL;DR

This paper introduces 3DMM-RF, a novel facial 3D Morphable Model that combines neural radiance fields with a style-based generator to improve accuracy, flexibility, and rendering speed for 3D face modeling and synthesis.

Contribution

It presents a style-based generative network that synthesizes neural radiance field samples in one pass, enabling fast, accurate, and controllable 3D face modeling from in-the-wild images.

Findings

Achieves accurate modeling of identity, pose, and expression.

Enables rendering in arbitrary illumination and pose.

Demonstrates generalization to real-world facial images.

Abstract

Facial 3D Morphable Models are a main computer vision subject with countless applications and have been highly optimized in the last two decades. The tremendous improvements of deep generative networks have created various possibilities for improving such models and have attracted wide interest. Moreover, the recent advances in neural radiance fields, are revolutionising novel-view synthesis of known scenes. In this work, we present a facial 3D Morphable Model, which exploits both of the above, and can accurately model a subject's identity, pose and expression and render it in arbitrary illumination. This is achieved by utilizing a powerful deep style-based generator to overcome two main weaknesses of neural radiance fields, their rigidity and rendering speed. We introduce a style-based generative network that synthesizes in one pass all and only the required rendering samples of a neural radiance field. We create a vast labelled synthetic dataset of facial renders, and train the network on these data, so that it can accurately model and generalize on facial identity, pose and appearance. Finally, we show that this model can accurately be fit to "in-the-wild" facial images of arbitrary pose and illumination, extract the facial characteristics, and be used to re-render the face in controllable conditions.