FaceCLIPNeRF: Text-driven 3D Face Manipulation using Deformable Neural Radiance Fields

TL;DR

FaceCLIPNeRF enables text-driven 3D face manipulation by combining deformable NeRFs with a novel spatially varying latent code system, allowing high-fidelity, user-friendly editing without manual masks.

Contribution

This work introduces a novel method for text-driven 3D face manipulation using deformable NeRFs and a spatially adaptive latent code system, eliminating the need for manual annotations.

Findings

Effective manipulation of 3D faces using text prompts.

Outperforms existing methods in quality and flexibility.

Demonstrates robustness across various face styles and expressions.

Abstract

As recent advances in Neural Radiance Fields (NeRF) have enabled high-fidelity 3D face reconstruction and novel view synthesis, its manipulation also became an essential task in 3D vision. However, existing manipulation methods require extensive human labor, such as a user-provided semantic mask and manual attribute search unsuitable for non-expert users. Instead, our approach is designed to require a single text to manipulate a face reconstructed with NeRF. To do so, we first train a scene manipulator, a latent code-conditional deformable NeRF, over a dynamic scene to control a face deformation using the latent code. However, representing a scene deformation with a single latent code is unfavorable for compositing local deformations observed in different instances. As so, our proposed Position-conditional Anchor Compositor (PAC) learns to represent a manipulated scene with spatially varying latent codes. Their renderings with the scene manipulator are then optimized to yield high cosine similarity to a target text in CLIP embedding space for text-driven manipulation. To the best of our knowledge, our approach is the first to address the text-driven manipulation of a face reconstructed with NeRF. Extensive results, comparisons, and ablation studies demonstrate the effectiveness of our approach.