DicFace: Dirichlet-Constrained Variational Codebook Learning for Temporally Coherent Video Face Restoration

TL;DR

DicFace introduces a Dirichlet-constrained variational codebook learning framework that leverages pretrained VQ-VAEs and spatio-temporal Transformers to achieve temporally coherent and high-quality video face restoration, effectively reducing flicker artifacts.

Contribution

The paper proposes a novel Dirichlet-distributed latent space and a spatio-temporal Transformer architecture for improved video face restoration, extending static image priors to video tasks.

Findings

Achieves state-of-the-art results on face restoration benchmarks.

Effectively reduces flicker artifacts in video sequences.

Demonstrates versatility across multiple video restoration tasks.

Abstract

Video face restoration faces a critical challenge in maintaining temporal consistency while recovering fine facial details from degraded inputs. This paper presents a novel approach that extends Vector-Quantized Variational Autoencoders (VQ-VAEs), pretrained on static high-quality portraits, into a video restoration framework through variational latent space modeling. Our key innovation lies in reformulating discrete codebook representations as Dirichlet-distributed continuous variables, enabling probabilistic transitions between facial features across frames. A spatio-temporal Transformer architecture jointly models inter-frame dependencies and predicts latent distributions, while a Laplacian-constrained reconstruction loss combined with perceptual (LPIPS) regularization enhances both pixel accuracy and visual quality. Comprehensive evaluations on blind face restoration, video inpainting, and facial colorization tasks demonstrate state-of-the-art performance. This work establishes an effective paradigm for adapting intensive image priors, pretrained on high-quality images, to video restoration while addressing the critical challenge of flicker artifacts. The source code has been open-sourced and is available at https://github.com/fudan-generative-vision/DicFace.