Learning to Decouple the Lights for 3D Face Texture Modeling

TL;DR

This paper presents a novel neural framework called Light Decoupling that models 3D facial textures under complex, occlusion-affected illumination by decomposing it into multiple light conditions, improving texture reconstruction accuracy.

Contribution

It introduces a new method to decouple complex illumination into multiple components for better 3D face texture modeling under challenging conditions.

Findings

Effective in modeling facial textures with occlusions and complex lighting

Outperforms existing methods on single images and video sequences

Demonstrates robustness to unnatural illumination scenarios

Abstract

Existing research has made impressive strides in reconstructing human facial shapes and textures from images with well-illuminated faces and minimal external occlusions. Nevertheless, it remains challenging to recover accurate facial textures from scenarios with complicated illumination affected by external occlusions, e.g. a face that is partially obscured by items such as a hat. Existing works based on the assumption of single and uniform illumination cannot correctly process these data. In this work, we introduce a novel approach to model 3D facial textures under such unnatural illumination. Instead of assuming single illumination, our framework learns to imitate the unnatural illumination as a composition of multiple separate light conditions combined with learned neural representations, named Light Decoupling. According to experiments on both single images and video sequences, we demonstrate the effectiveness of our approach in modeling facial textures under challenging illumination affected by occlusions. Please check https://tianxinhuang.github.io/projects/Deface for our videos and codes.