FPGA: Flexible Portrait Generation Approach

TL;DR

FPGA is a versatile portrait generation system that improves multi-ID face synthesis, reduces artifacts, and accelerates inference, using a novel training strategy and a plug-and-play restoration framework.

Contribution

The paper introduces FPGA, a comprehensive portrait generation approach with a new training strategy and a restoration framework that enhances multi-ID synthesis and artifact reduction.

Findings

Significant improvements in subjective and objective metrics.

Achieves controllable multi-ID portrait generation.

Inference speed within 2.5 seconds on a single GPU.

Abstract

Portrait Fidelity Generation is a prominent research area in generative models.Current methods face challenges in generating full-body images with low-resolution faces, especially in multi-ID photo phenomenon.To tackle these issues, we propose a comprehensive system called FPGA and construct a million-level multi-modal dataset IDZoom for training.FPGA consists of Multi-Mode Fusion training strategy (MMF) and DDIM Inversion based ID Restoration inference framework (DIIR). The MMF aims to activate the specified ID in the specified facial region. The DIIR aims to address the issue of face artifacts while keeping the background.Furthermore, DIIR is plug-and-play and can be applied to any diffusion-based portrait generation method to enhance their performance. DIIR is also capable of performing face-swapping tasks and is applicable to stylized faces as well.To validate the effectiveness of FPGA, we conducted extensive comparative and ablation experiments. The experimental results demonstrate that FPGA has significant advantages in both subjective and objective metrics, and achieves controllable generation in multi-ID scenarios. In addition, we accelerate the inference speed to within 2.5 seconds on a single L20 graphics card mainly based on our well designed reparameterization method, RepControlNet.