Controllable 3D Face Generation with Conditional Style Code Diffusion

TL;DR

This paper introduces TEx-Face, a novel method for efficient, controllable 3D face generation that combines GAN inversion, style code diffusion, and face decoding, addressing existing inefficiencies and control limitations.

Contribution

The paper presents a new framework dividing 3D face generation into three components, with innovations in style code enhancement, denoising, and data augmentation for improved control and efficiency.

Findings

Achieves photorealistic 3D face generation with high controllability.

Demonstrates superior performance on FFHQ, CelebA-HQ, and CelebA-Dialog datasets.

Provides a scalable approach with publicly available code.

Abstract

Generating photorealistic 3D faces from given conditions is a challenging task. Existing methods often rely on time-consuming one-by-one optimization approaches, which are not efficient for modeling the same distribution content, e.g., faces. Additionally, an ideal controllable 3D face generation model should consider both facial attributes and expressions. Thus we propose a novel approach called TEx-Face(TExt & Expression-to-Face) that addresses these challenges by dividing the task into three components, i.e., 3D GAN Inversion, Conditional Style Code Diffusion, and 3D Face Decoding. For 3D GAN inversion, we introduce two methods which aim to enhance the representation of style codes and alleviate 3D inconsistencies. Furthermore, we design a style code denoiser to incorporate multiple conditions into the style code and propose a data augmentation strategy to address the issue of insufficient paired visual-language data. Extensive experiments conducted on FFHQ, CelebA-HQ, and CelebA-Dialog demonstrate the promising performance of our TEx-Face in achieving the efficient and controllable generation of photorealistic 3D faces. The code will be available at https://github.com/sxl142/TEx-Face.