Towards Consistent and Controllable Image Synthesis for Face Editing

TL;DR

This paper introduces RigFace, a novel face editing method that leverages Stable-Diffusion and 3D face models to achieve consistent, controllable, and high-quality face edits by disentangling background, pose, expression, and lighting controls.

Contribution

The paper presents a new approach combining a Spatial Attribute Encoder, FaceFusion, and Attribute Rigger to improve control and consistency in face editing with diffusion models.

Findings

Achieves superior identity preservation and photorealism.

Effectively disentangles control of background, pose, expression, and lighting.

Outperforms existing face editing methods in key metrics.

Abstract

Face editing methods, essential for tasks like virtual avatars, digital human synthesis and identity preservation, have traditionally been built upon GAN-based techniques, while recent focus has shifted to diffusion-based models due to their success in image reconstruction. However, diffusion models still face challenges in controlling specific attributes and preserving the consistency of other unchanged attributes especially the identity characteristics. To address these issues and facilitate more convenient editing of face images, we propose a novel approach that leverages the power of Stable-Diffusion (SD) models and crude 3D face models to control the lighting, facial expression and head pose of a portrait photo. We observe that this task essentially involves the combinations of target background, identity and face attributes aimed to edit. We strive to sufficiently disentangle the control of these factors to enable consistency of face editing. Specifically, our method, coined as RigFace, contains: 1) A Spatial Attribute Encoder that provides presise and decoupled conditions of background, pose, expression and lighting; 2) A high-consistency FaceFusion method that transfers identity features from the Identity Encoder to the denoising UNet of a pre-trained SD model; 3) An Attribute Rigger that injects those conditions into the denoising UNet. Our model achieves comparable or even superior performance in both identity preservation and photorealism compared to existing face editing models.