Controllable Face Manipulation and UV Map Generation by Self-supervised Learning

TL;DR

This paper introduces a self-supervised method to control facial attributes and generate high-quality UV maps by editing StyleGAN's latent space with 3DMM parameters, addressing domain gap issues.

Contribution

It proposes a novel 'Map and edit' network to improve explicit face attribute control and UV texture generation without manual annotations.

Findings

Accurately generates multi-view face images with consistent identity.

Produces high-resolution, texture-rich UV facial textures.

Avoids domain gap issues with a new attribute editing method.

Abstract

Although manipulating facial attributes by Generative Adversarial Networks (GANs) has been remarkably successful recently, there are still some challenges in explicit control of features such as pose, expression, lighting, etc. Recent methods achieve explicit control over 2D images by combining 2D generative model and 3DMM. However, due to the lack of realism and clarity in texture reconstruction by 3DMM, there is a domain gap between the synthetic image and the rendered image of 3DMM. Since rendered 3DMM images contain facial region only without the background, directly computing the loss between these two domains is not ideal and the resultant trained model will be biased. In this study, we propose to explicitly edit the latent space of the pretrained StyleGAN by controlling the parameters of the 3DMM. To address the domain gap problem, we propose a noval network called 'Map and edit' and a simple but effective attribute editing method to avoid direct loss computation between rendered and synthesized images. Furthermore, since our model can accurately generate multi-view face images while the identity remains unchanged. As a by-product, combined with visibility masks, our proposed model can also generate texture-rich and high-resolution UV facial textures. Our model relies on pretrained StyleGAN, and the proposed model is trained in a self-supervised manner without any manual annotations or datasets.