Monocular Reconstruction of Neural Face Reflectance Fields

TL;DR

This paper introduces a neural method for reconstructing detailed face reflectance fields from a single image, capturing complex lighting effects like specularities and shadows, enabling photorealistic rendering from any viewpoint and lighting.

Contribution

A novel neural representation that estimates comprehensive face reflectance components from monocular images, surpassing prior methods in realism and physical accuracy.

Findings

Outperforms existing methods in photorealism

Captures complex lighting effects including specularities and shadows

Enables rendering from arbitrary viewpoints and lighting conditions

Abstract

The reflectance field of a face describes the reflectance properties responsible for complex lighting effects including diffuse, specular, inter-reflection and self shadowing. Most existing methods for estimating the face reflectance from a monocular image assume faces to be diffuse with very few approaches adding a specular component. This still leaves out important perceptual aspects of reflectance as higher-order global illumination effects and self-shadowing are not modeled. We present a new neural representation for face reflectance where we can estimate all components of the reflectance responsible for the final appearance from a single monocular image. Instead of modeling each component of the reflectance separately using parametric models, our neural representation allows us to generate a basis set of faces in a geometric deformation-invariant space, parameterized by the input light direction, viewpoint and face geometry. We learn to reconstruct this reflectance field of a face just from a monocular image, which can be used to render the face from any viewpoint in any light condition. Our method is trained on a light-stage training dataset, which captures 300 people illuminated with 150 light conditions from 8 viewpoints. We show that our method outperforms existing monocular reflectance reconstruction methods, in terms of photorealism due to better capturing of physical premitives, such as sub-surface scattering, specularities, self-shadows and other higher-order effects.