Beyond Alignment: Blind Video Face Restoration via Parsing-Guided Temporal-Coherent Transformer

TL;DR

This paper introduces PGTFormer, a novel blind video face restoration method that uses semantic parsing guidance and a transformer architecture to produce temporally coherent, high-quality face videos without pre-alignment.

Contribution

It presents the first blind video face restoration approach with a parsing-guided transformer that avoids pre-alignment and reduces artifacts, improving temporal coherence and fidelity.

Findings

Outperforms previous face restoration methods in experiments.

Effectively reduces artifacts and jitter in restored videos.

Enhances temporal consistency without face pre-alignment.

Abstract

Multiple complex degradations are coupled in low-quality video faces in the real world. Therefore, blind video face restoration is a highly challenging ill-posed problem, requiring not only hallucinating high-fidelity details but also enhancing temporal coherence across diverse pose variations. Restoring each frame independently in a naive manner inevitably introduces temporal incoherence and artifacts from pose changes and keypoint localization errors. To address this, we propose the first blind video face restoration approach with a novel parsing-guided temporal-coherent transformer (PGTFormer) without pre-alignment. PGTFormer leverages semantic parsing guidance to select optimal face priors for generating temporally coherent artifact-free results. Specifically, we pre-train a temporal-spatial vector quantized auto-encoder on high-quality video face datasets to extract expressive context-rich priors. Then, the temporal parse-guided codebook predictor (TPCP) restores faces in different poses based on face parsing context cues without performing face pre-alignment. This strategy reduces artifacts and mitigates jitter caused by cumulative errors from face pre-alignment. Finally, the temporal fidelity regulator (TFR) enhances fidelity through temporal feature interaction and improves video temporal consistency. Extensive experiments on face videos show that our method outperforms previous face restoration baselines. The code will be released on \href{https://github.com/kepengxu/PGTFormer}{https://github.com/kepengxu/PGTFormer}.