3D-GANTex: 3D Face Reconstruction with StyleGAN3-based Multi-View Images and 3DDFA based Mesh Generation

TL;DR

This paper presents a novel approach for 3D face reconstruction from a single image by combining StyleGAN's multi-view face generation with 3DDFA's mesh and texture estimation, achieving high-quality results.

Contribution

It introduces a new method that integrates StyleGAN and 3DDFA to improve 3D face reconstruction from single images, especially for rotated faces.

Findings

High-quality 3D face meshes generated

Near accurate texture representations achieved

Effective multi-view face synthesis from a single image

Abstract

Geometry and texture estimation from a single face image is an ill-posed problem since there is very little information to work with. The problem further escalates when the face is rotated at a different angle. This paper tries to tackle this problem by introducing a novel method for texture estimation from a single image by first using StyleGAN and 3D Morphable Models. The method begins by generating multi-view faces using the latent space of GAN. Then 3DDFA trained on 3DMM estimates a 3D face mesh as well as a high-resolution texture map that is consistent with the estimated face shape. The result shows that the generated mesh is of high quality with near to accurate texture representation.