Self-supervised Re-renderable Facial Albedo Reconstruction from Single Image

TL;DR

This paper introduces a self-supervised deep learning framework for reconstructing high-quality, re-renderable 3D facial albedos from single images, effectively disentangling illumination and texture without ground-truth data.

Contribution

It proposes a novel combination of a 3DMM-based prior, detail refinement, and a detailed illumination representation with regularization losses for improved facial texture reconstruction.

Findings

Outperforms state-of-the-art methods on challenging datasets.

Achieves high-fidelity, re-renderable facial textures with disentangled illumination.

Enables self-supervised training without ground-truth facial reflectance data.

Abstract

Reconstructing high-fidelity 3D facial texture from a single image is a quite challenging task due to the lack of complete face information and the domain gap between the 3D face and 2D image. Further, obtaining re-renderable 3D faces has become a strongly desired property in many applications, where the term 're-renderable' demands the facial texture to be spatially complete and disentangled with environmental illumination. In this paper, we propose a new self-supervised deep learning framework for reconstructing high-quality and re-renderable facial albedos from single-view images in-the-wild. Our main idea is to first utilize a prior generation module based on the 3DMM proxy model to produce an unwrapped texture and a globally parameterized prior albedo. Then we apply a detail refinement module to synthesize the final texture with both high-frequency details and completeness. To further make facial textures disentangled with illumination, we propose a novel detailed illumination representation which is reconstructed with the detailed albedo together. We also design several novel regularization losses on both the albedo and illumination maps to facilitate the disentanglement of these two factors. Finally, by leveraging a differentiable renderer, each face attribute can be jointly trained in a self-supervised manner without requiring ground-truth facial reflectance. Extensive comparisons and ablation studies on challenging datasets demonstrate that our framework outperforms state-of-the-art approaches.