Fake It Without Making It: Conditioned Face Generation for Accurate 3D Face Reconstruction

TL;DR

This paper introduces SynthFace, a large-scale synthetic dataset of photorealistic faces generated via conditioned diffusion, and ControlFace, a neural network trained on this data for accurate 3D face reconstruction without 3D supervision.

Contribution

The paper presents SynthFace, a diverse synthetic dataset generated by conditioning diffusion models on 3D shape parameters, and ControlFace, a novel neural network trained on SynthFace for improved 3D face reconstruction.

Findings

ControlFace achieves competitive results on NoW benchmark.

SynthFace dataset is diverse, balanced, and publicly available.

Method reduces reliance on limited 3D data for training.

Abstract

Accurate 3D face reconstruction from 2D images is an enabling technology with applications in healthcare, security, and creative industries. However, current state-of-the-art methods either rely on supervised training with very limited 3D data or self-supervised training with 2D image data. To bridge this gap, we present a method to generate a large-scale synthesised dataset of 250K photorealistic images and their corresponding shape parameters and depth maps, which we call SynthFace. Our synthesis method conditions Stable Diffusion on depth maps sampled from the FLAME 3D Morphable Model (3DMM) of the human face, allowing us to generate a diverse set of shape-consistent facial images that is designed to be balanced in race and gender. We further propose ControlFace, a deep neural network, trained on SynthFace, which achieves competitive performance on the NoW benchmark, without requiring 3D supervision or manual 3D asset creation. The complete SynthFace dataset will be made publicly available upon publication.