AvatarMe++: Facial Shape and BRDF Inference with Photorealistic Rendering-Aware GANs

TL;DR

This paper introduces a novel method for reconstructing high-resolution, photorealistic 3D facial geometry and BRDF from a single in-the-wild image, enabling realistic rendering and applications in face analysis.

Contribution

It presents the first approach capable of producing render-ready 3D faces from in-the-wild images, utilizing a new dataset and a photorealistic differentiable rendering methodology.

Findings

Outperforms existing methods significantly

Reconstructs high-resolution 3D faces from low-resolution images

Enables realistic rendering and application in various domains

Abstract

Over the last years, many face analysis tasks have accomplished astounding performance, with applications including face generation and 3D face reconstruction from a single "in-the-wild" image. Nevertheless, to the best of our knowledge, there is no method which can produce render-ready high-resolution 3D faces from "in-the-wild" images and this can be attributed to the: (a) scarcity of available data for training, and (b) lack of robust methodologies that can successfully be applied on very high-resolution data. In this work, we introduce the first method that is able to reconstruct photorealistic render-ready 3D facial geometry and BRDF from a single "in-the-wild" image. We capture a large dataset of facial shape and reflectance, which we have made public. We define a fast facial photorealistic differentiable rendering methodology with accurate facial skin diffuse and specular reflection, self-occlusion and subsurface scattering approximation. With this, we train a network that disentangles the facial diffuse and specular BRDF components from a shape and texture with baked illumination, reconstructed with a state-of-the-art 3DMM fitting method. Our method outperforms the existing arts by a significant margin and reconstructs high-resolution 3D faces from a single low-resolution image, that can be rendered in various applications, and bridge the uncanny valley.