High-Fidelity 3D Facial Avatar Synthesis with Controllable Fine-Grained Expressions

TL;DR

This paper introduces a novel method for high-fidelity 3D facial avatar synthesis that enables precise control over fine-grained expressions by refining texture and mesh editing through a dual mapper and text-guided optimization.

Contribution

It proposes a dual mapper framework with a CLIP-based text-guided optimization to achieve fine-grained expression control in 3D facial synthesis, addressing limitations of previous methods.

Findings

Effective control of fine-grained facial expressions demonstrated

Superior quality and view consistency in generated avatars

Outperforms existing methods in experiments

Abstract

Facial expression editing methods can be mainly categorized into two types based on their architectures: 2D-based and 3D-based methods. The former lacks 3D face modeling capabilities, making it difficult to edit 3D factors effectively. The latter has demonstrated superior performance in generating high-quality and view-consistent renderings using single-view 2D face images. Although these methods have successfully used animatable models to control facial expressions, they still have limitations in achieving precise control over fine-grained expressions. To address this issue, in this paper, we propose a novel approach by simultaneously refining both the latent code of a pretrained 3D-Aware GAN model for texture editing and the expression code of the driven 3DMM model for mesh editing. Specifically, we introduce a Dual Mappers module, comprising Texture Mapper and Emotion Mapper, to learn the transformations of the given latent code for textures and the expression code for meshes, respectively. To optimize the Dual Mappers, we propose a Text-Guided Optimization method, leveraging a CLIP-based objective function with expression text prompts as targets, while integrating a SubSpace Projection mechanism to project the text embedding to the expression subspace such that we can have more precise control over fine-grained expressions. Extensive experiments and comparative analyses demonstrate the effectiveness and superiority of our proposed method.