ControlFace: Harnessing Facial Parametric Control for Face Rigging

TL;DR

ControlFace introduces a novel face rigging method that uses 3DMM renderings and dual-branch U-Nets to achieve high-fidelity, identity-preserving facial manipulation with precise control, overcoming limitations of dataset-dependent approaches.

Contribution

The paper presents a new face rigging technique conditioned on 3DMM renderings, featuring a dual-branch U-Net architecture and a control mixer module for enhanced control accuracy and identity preservation.

Findings

Outperforms existing methods in identity preservation

Achieves higher control accuracy in facial manipulation

Demonstrates practical applicability through extensive experiments

Abstract

Manipulation of facial images to meet specific controls such as pose, expression, and lighting, also known as face rigging, is a complex task in computer vision. Existing methods are limited by their reliance on image datasets, which necessitates individual-specific fine-tuning and limits their ability to retain fine-grained identity and semantic details, reducing practical usability. To overcome these limitations, we introduce ControlFace, a novel face rigging method conditioned on 3DMM renderings that enables flexible, high-fidelity control. We employ a dual-branch U-Nets: one, referred to as FaceNet, captures identity and fine details, while the other focuses on generation. To enhance control precision, the control mixer module encodes the correlated features between the target-aligned control and reference-aligned control, and a novel guidance method, reference control guidance, steers the generation process for better control adherence. By training on a facial video dataset, we fully utilize FaceNet's rich representations while ensuring control adherence. Extensive experiments demonstrate ControlFace's superior performance in identity preservation and control precision, highlighting its practicality. Please see the project website: https://cvlab-kaist.github.io/ControlFace/.