Unsupervised High-Fidelity Facial Texture Generation and Reconstruction

TL;DR

This paper introduces a novel unsupervised framework for high-fidelity 3D facial texture generation and reconstruction, leveraging 2D facial generation techniques to improve realism and diversity without relying on scanned textures.

Contribution

It presents the first unified, unsupervised pipeline for both 3D facial geometry and texture generation, integrating 2D GANs with 3D modeling for enhanced realism.

Findings

Achieves state-of-the-art results in facial texture reconstruction.

Demonstrates high realism and diversity in generated 3D faces.

Outperforms recent methods in both generation and reconstruction tasks.

Abstract

Many methods have been proposed over the years to tackle the task of facial 3D geometry and texture recovery from a single image. Such methods often fail to provide high-fidelity texture without relying on 3D facial scans during training. In contrast, the complementary task of 3D facial generation has not received as much attention. As opposed to the 2D texture domain, where GANs have proven to produce highly realistic facial images, the more challenging 3D geometry domain has not yet caught up to the same levels of realism and diversity. In this paper, we propose a novel unified pipeline for both tasks, generation of both geometry and texture, and recovery of high-fidelity texture. Our texture model is learned, in an unsupervised fashion, from natural images as opposed to scanned texture maps. To the best of our knowledge, this is the first such unified framework independent of scanned textures. Our novel training pipeline incorporates a pre-trained 2D facial generator coupled with a deep feature manipulation methodology. By applying precise 3DMM fitting, we can seamlessly integrate our modeled textures into synthetically generated background images forming a realistic composition of our textured model with background, hair, teeth, and body. This enables us to apply transfer learning from the domain of 2D image generation, thus, benefiting greatly from the impressive results obtained in this domain. We provide a comprehensive study on several recent methods comparing our model in generation and reconstruction tasks. As the extensive qualitative, as well as quantitative analysis, demonstrate, we achieve state-of-the-art results for both tasks.