NeRFlame: FLAME-based conditioning of NeRF for 3D face rendering

TL;DR

NeRFlame combines FLAME's controllability with NeRF's rendering quality by integrating FLAME meshes into the NeRF architecture, enabling high-quality, controllable 3D face rendering.

Contribution

The paper introduces NeRFlame, a novel method that integrates FLAME mesh models into NeRF to improve controllability and detail in 3D face rendering.

Findings

Enables high-quality face rendering with explicit control.

Uses FLAME mesh as a density volume for better geometry control.

Achieves detailed and realistic face images with controllable expressions.

Abstract

Traditional 3D face models are based on mesh representations with texture. One of the most important models is FLAME (Faces Learned with an Articulated Model and Expressions), which produces meshes of human faces that are fully controllable. Unfortunately, such models have problems with capturing geometric and appearance details. In contrast to mesh representation, the neural radiance field (NeRF) produces extremely sharp renders. However, implicit methods are hard to animate and do not generalize well to unseen expressions. It is not trivial to effectively control NeRF models to obtain face manipulation. The present paper proposes a novel approach, named NeRFlame, which combines the strengths of both NeRF and FLAME methods. Our method enables high-quality rendering capabilities of NeRF while also offering complete control over the visual appearance, similar to FLAME. In contrast to traditional NeRF-based structures that use neural networks for RGB color and volume density modeling, our approach utilizes the FLAME mesh as a distinct density volume. Consequently, color values exist only in the vicinity of the FLAME mesh. This FLAME framework is seamlessly incorporated into the NeRF architecture for predicting RGB colors, enabling our model to explicitly represent volume density and implicitly capture RGB colors.