HiFi-Portrait: Zero-shot Identity-preserved Portrait Generation with High-fidelity Multi-face Fusion

TL;DR

HiFi-Portrait is a novel zero-shot portrait generation method that achieves high fidelity and precise face attribute control by fusing multi-face features and landmarks, surpassing state-of-the-art approaches.

Contribution

The paper introduces HiFi-Portrait, a new high-fidelity, zero-shot portrait generation framework with multi-face fusion and landmark-guided face control, including an automated dataset construction pipeline.

Findings

Outperforms SOTA in face similarity and controllability.

Compatible with SDXL-based models.

Achieves high-fidelity, identity-preserved multi-face portrait generation.

Abstract

Recent advancements in diffusion-based technologies have made significant strides, particularly in identity-preserved portrait generation (IPG). However, when using multiple reference images from the same ID, existing methods typically produce lower-fidelity portraits and struggle to customize face attributes precisely. To address these issues, this paper presents HiFi-Portrait, a high-fidelity method for zero-shot portrait generation. Specifically, we first introduce the face refiner and landmark generator to obtain fine-grained multi-face features and 3D-aware face landmarks. The landmarks include the reference ID and the target attributes. Then, we design HiFi-Net to fuse multi-face features and align them with landmarks, which improves ID fidelity and face control. In addition, we devise an automated pipeline to construct an ID-based dataset for training HiFi-Portrait. Extensive experimental results demonstrate that our method surpasses the SOTA approaches in face similarity and controllability. Furthermore, our method is also compatible with previous SDXL-based works.