Face Relighting with Geometrically Consistent Shadows

TL;DR

This paper introduces a differentiable ray tracing algorithm for face relighting that accurately synthesizes hard shadows using face geometry, leading to more realistic images and improved geometry estimation.

Contribution

We propose a novel differentiable shadow synthesis method based on ray tracing that leverages face geometry for consistent hard shadow rendering in relighting.

Findings

Our method outperforms previous approaches in shadow realism.

It achieves state-of-the-art face relighting performance.

Improves face geometry estimation quality.

Abstract

Most face relighting methods are able to handle diffuse shadows, but struggle to handle hard shadows, such as those cast by the nose. Methods that propose techniques for handling hard shadows often do not produce geometrically consistent shadows since they do not directly leverage the estimated face geometry while synthesizing them. We propose a novel differentiable algorithm for synthesizing hard shadows based on ray tracing, which we incorporate into training our face relighting model. Our proposed algorithm directly utilizes the estimated face geometry to synthesize geometrically consistent hard shadows. We demonstrate through quantitative and qualitative experiments on Multi-PIE and FFHQ that our method produces more geometrically consistent shadows than previous face relighting methods while also achieving state-of-the-art face relighting performance under directional lighting. In addition, we demonstrate that our differentiable hard shadow modeling improves the quality of the estimated face geometry over diffuse shading models.