FaceMe: Robust Blind Face Restoration with Personal Identification

TL;DR

FaceMe is a novel diffusion-based method for blind face restoration that uses identity features from reference images to produce high-quality, identity-preserving facial images without fine-tuning during inference.

Contribution

We introduce a personalized face restoration approach leveraging identity encoders and diffusion models, supporting multiple references and robust identity preservation without fine-tuning.

Findings

Restores high-quality, identity-consistent facial images

Supports multiple reference images during inference

Demonstrates robustness and excellent performance

Abstract

Blind face restoration is a highly ill-posed problem due to the lack of necessary context. Although existing methods produce high-quality outputs, they often fail to faithfully preserve the individual's identity. In this paper, we propose a personalized face restoration method, FaceMe, based on a diffusion model. Given a single or a few reference images, we use an identity encoder to extract identity-related features, which serve as prompts to guide the diffusion model in restoring high-quality and identity-consistent facial images. By simply combining identity-related features, we effectively minimize the impact of identity-irrelevant features during training and support any number of reference image inputs during inference. Additionally, thanks to the robustness of the identity encoder, synthesized images can be used as reference images during training, and identity changing during inference does not require fine-tuning the model. We also propose a pipeline for constructing a reference image training pool that simulates the poses and expressions that may appear in real-world scenarios. Experimental results demonstrate that our FaceMe can restore high-quality facial images while maintaining identity consistency, achieving excellent performance and robustness.