Practical Face Reconstruction via Differentiable Ray Tracing

TL;DR

This paper introduces a novel differentiable ray-tracing method for face reconstruction from monocular images, accurately estimating geometry, reflectance, pose, and illumination even under challenging lighting conditions.

Contribution

It proposes a new coarse-to-fine optimization framework with a parameterized virtual light stage for comprehensive face attribute estimation from unconstrained images.

Findings

Effective under extreme illumination conditions

Outperforms recent state-of-the-art methods in accuracy

Enables versatile face editing and transfer applications

Abstract

We present a differentiable ray-tracing based novel face reconstruction approach where scene attributes - 3D geometry, reflectance (diffuse, specular and roughness), pose, camera parameters, and scene illumination - are estimated from unconstrained monocular images. The proposed method models scene illumination via a novel, parameterized virtual light stage, which in-conjunction with differentiable ray-tracing, introduces a coarse-to-fine optimization formulation for face reconstruction. Our method can not only handle unconstrained illumination and self-shadows conditions, but also estimates diffuse and specular albedos. To estimate the face attributes consistently and with practical semantics, a two-stage optimization strategy systematically uses a subset of parametric attributes, where subsequent attribute estimations factor those previously estimated. For example, self-shadows estimated during the first stage, later prevent its baking into the personalized diffuse and specular albedos in the second stage. We show the efficacy of our approach in several real-world scenarios, where face attributes can be estimated even under extreme illumination conditions. Ablation studies, analyses and comparisons against several recent state-of-the-art methods show improved accuracy and versatility of our approach. With consistent face attributes reconstruction, our method leads to several style -- illumination, albedo, self-shadow -- edit and transfer applications, as discussed in the paper.