A Generalist FaceX via Learning Unified Facial Representation

TL;DR

This paper introduces FaceX, a unified facial model capable of handling multiple facial editing tasks simultaneously by decomposing faces into fundamental components and leveraging diffusion models for high-quality generation.

Contribution

The work proposes a novel unified facial representation and a framework that integrates diffusion priors for versatile and efficient facial editing across diverse tasks.

Findings

Achieves competitive performance on facial editing tasks

Uses diffusion priors for improved generation quality

Provides a unified approach to multiple facial tasks

Abstract

This work presents FaceX framework, a novel facial generalist model capable of handling diverse facial tasks simultaneously. To achieve this goal, we initially formulate a unified facial representation for a broad spectrum of facial editing tasks, which macroscopically decomposes a face into fundamental identity, intra-personal variation, and environmental factors. Based on this, we introduce Facial Omni-Representation Decomposing (FORD) for seamless manipulation of various facial components, microscopically decomposing the core aspects of most facial editing tasks. Furthermore, by leveraging the prior of a pretrained StableDiffusion (SD) to enhance generation quality and accelerate training, we design Facial Omni-Representation Steering (FORS) to first assemble unified facial representations and then effectively steer the SD-aware generation process by the efficient Facial Representation Controller (FRC). %Without any additional features, Our versatile FaceX achieves competitive performance compared to elaborate task-specific models on popular facial editing tasks. Full codes and models will be available at https://github.com/diffusion-facex/FaceX.