FLAIR: A Conditional Diffusion Framework with Applications to Face Video Restoration

TL;DR

FLAIR is a novel conditional diffusion framework designed for face video restoration that maintains temporal consistency and high perceptual quality, outperforming current state-of-the-art methods across various tasks.

Contribution

The paper introduces FLAIR, a new diffusion-based method that converts image DPMs into video DPMs with temporal coherence, using recurrent layers and self-attention, and incorporates a refinement process for enhanced face video restoration.

Findings

Outperforms SOTA in face video super-resolution, deblurring, JPEG restoration, and interpolation.

Ensures temporal consistency with a recurrent refinement layer and self-attention.

Achieves high perceptual quality while maintaining fidelity to degraded inputs.

Abstract

Face video restoration (FVR) is a challenging but important problem where one seeks to recover a perceptually realistic face videos from a low-quality input. While diffusion probabilistic models (DPMs) have been shown to achieve remarkable performance for face image restoration, they often fail to preserve temporally coherent, high-quality videos, compromising the fidelity of reconstructed faces. We present a new conditional diffusion framework called FLAIR for FVR. FLAIR ensures temporal consistency across frames in a computationally efficient fashion by converting a traditional image DPM into a video DPM. The proposed conversion uses a recurrent video refinement layer and a temporal self-attention at different scales. FLAIR also uses a conditional iterative refinement process to balance the perceptual and distortion quality during inference. This process consists of two key components: a data-consistency module that analytically ensures that the generated video precisely matches its degraded observation and a coarse-to-fine image enhancement module specifically for facial regions. Our extensive experiments show superiority of FLAIR over the current state-of-the-art (SOTA) for video super-resolution, deblurring, JPEG restoration, and space-time frame interpolation on two high-quality face video datasets.