LAFR: Efficient Diffusion-based Blind Face Restoration via Latent Codebook Alignment Adapter

TL;DR

LAFR introduces a novel latent codebook alignment adapter for diffusion-based blind face restoration, significantly improving semantic consistency and identity preservation without retraining the VAE, while reducing training time.

Contribution

The paper proposes a codebook-based latent space adapter for diffusion models that aligns low-quality and high-quality image latents, enhancing face restoration efficiency and effectiveness.

Findings

Achieves high-quality, identity-preserving face restoration from degraded images.

Reduces training time by 70% with minimal finetuning on FFHQ dataset.

Outperforms existing methods on synthetic and real-world benchmarks.

Abstract

Blind face restoration from low-quality (LQ) images is a challenging task that requires not only high-fidelity image reconstruction but also the preservation of facial identity. While diffusion models like Stable Diffusion have shown promise in generating high-quality (HQ) images, their VAE modules are typically trained only on HQ data, resulting in semantic misalignment when encoding LQ inputs. This mismatch significantly weakens the effectiveness of LQ conditions during the denoising process. Existing approaches often tackle this issue by retraining the VAE encoder, which is computationally expensive and memory-intensive. To address this limitation efficiently, we propose LAFR (Latent Alignment for Face Restoration), a novel codebook-based latent space adapter that aligns the latent distribution of LQ images with that of HQ counterparts, enabling semantically consistent diffusion sampling without altering the original VAE. To further enhance identity preservation, we introduce a multi-level restoration loss that combines constraints from identity embeddings and facial structural priors. Additionally, by leveraging the inherent structural regularity of facial images, we show that lightweight finetuning of diffusion prior on just 0.9% of FFHQ dataset is sufficient to achieve results comparable to state-of-the-art methods, reduce training time by 70%. Extensive experiments on both synthetic and real-world face restoration benchmarks demonstrate the effectiveness and efficiency of LAFR, achieving high-quality, identity-preserving face reconstruction from severely degraded inputs.