Face-MakeUpV2: Facial Consistency Learning for Controllable Text-to-Image Generation

TL;DR

Face-MakeUpV2 is a novel facial image generation model that ensures facial identity and physical consistency with reference images, addressing attribute leakage and local semantic instruction challenges.

Contribution

It introduces a large-scale dataset and a dual-channel injection method with optimized training objectives for improved facial consistency in text-to-image generation.

Findings

Achieves superior face ID preservation

Maintains physical consistency with reference images

Outperforms existing models in facial editing tasks

Abstract

In facial image generation, current text-to-image models often suffer from facial attribute leakage and insufficient physical consistency when responding to local semantic instructions. In this study, we propose Face-MakeUpV2, a facial image generation model that aims to maintain the consistency of face ID and physical characteristics with the reference image. First, we constructed a large-scale dataset FaceCaptionMask-1M comprising approximately one million image-text-masks pairs that provide precise spatial supervision for the local semantic instructions. Second, we employed a general text-to-image pretrained model as the backbone and introduced two complementary facial information injection channels: a 3D facial rendering channel to incorporate the physical characteristics of the image and a global facial feature channel. Third, we formulated two optimization objectives for the supervised learning of our model: semantic alignment in the model's embedding space to mitigate the attribute leakage problem and perceptual loss on facial images to preserve ID consistency. Extensive experiments demonstrated that our Face-MakeUpV2 achieves best overall performance in terms of preserving face ID and maintaining physical consistency of the reference images. These results highlight the practical potential of Face-MakeUpV2 for reliable and controllable facial editing in diverse applications.