Face Reflectance and Geometry Modeling via Differentiable Ray Tracing

TL;DR

This paper introduces a differentiable ray tracing approach for automatic 3D face reconstruction from monocular images, enabling explicit disentanglement of geometry, reflectance, and shadows for enhanced control and applications.

Contribution

It proposes a novel method combining differentiable Monte-Carlo ray tracing with virtual light staging to accurately reconstruct and disentangle facial properties from single images.

Findings

Robust facial geometry reconstruction achieved.

Explicit control over reflectance and shadows demonstrated.

Applications include expression editing and relighting.

Abstract

We present a novel strategy to automatically reconstruct 3D faces from monocular images with explicitly disentangled facial geometry (pose, identity and expression), reflectance (diffuse and specular albedo), and self-shadows. The scene lights are modeled as a virtual light stage with pre-oriented area lights used in conjunction with differentiable Monte-Carlo ray tracing to optimize the scene and face parameters. With correctly disentangled self-shadows and specular reflection parameters, we can not only obtain robust facial geometry reconstruction, but also gain explicit control over these parameters, with several practical applications. We can change facial expressions with accurate resultant self-shadows or relight the scene and obtain accurate specular reflection and several other parameter combinations.