ReF-LDM: A Latent Diffusion Model for Reference-based Face Image Restoration

TL;DR

ReF-LDM is a novel latent diffusion model that enhances face image restoration by conditioning on reference images and introduces CacheKV and identity loss mechanisms, supported by a new dataset for training and evaluation.

Contribution

The paper presents ReF-LDM, a new model for reference-based face restoration, with innovative mechanisms and a dedicated dataset, improving the fidelity and identity preservation of generated images.

Findings

ReF-LDM outperforms existing methods in face restoration quality.

The CacheKV mechanism effectively leverages reference images during generation.

The FFHQ-Ref dataset facilitates training and benchmarking of reference-based face restoration models.

Abstract

While recent works on blind face image restoration have successfully produced impressive high-quality (HQ) images with abundant details from low-quality (LQ) input images, the generated content may not accurately reflect the real appearance of a person. To address this problem, incorporating well-shot personal images as additional reference inputs could be a promising strategy. Inspired by the recent success of the Latent Diffusion Model (LDM), we propose ReF-LDM, an adaptation of LDM designed to generate HQ face images conditioned on one LQ image and multiple HQ reference images. Our model integrates an effective and efficient mechanism, CacheKV, to leverage the reference images during the generation process. Additionally, we design a timestep-scaled identity loss, enabling our LDM-based model to focus on learning the discriminating features of human faces. Lastly, we construct FFHQ-Ref, a dataset consisting of 20,405 high-quality (HQ) face images with corresponding reference images, which can serve as both training and evaluation data for reference-based face restoration models.