Audio-Assisted Face Video Restoration with Temporal and Identity Complementary Learning

TL;DR

This paper introduces GAVN, a novel audio-assisted face video restoration network that leverages temporal and identity features, along with audio cues, to effectively restore various degraded face videos.

Contribution

The paper presents a new network that combines temporal and identity features with audio signals for comprehensive face video restoration, addressing multiple types of distortions.

Findings

Outperforms state-of-the-art methods in artifact removal, deblurring, and super-resolution.

Effectively utilizes audio and face landmarks for detailed facial restoration.

Achieves high-quality face video reconstruction with reduced computational cost.

Abstract

Face videos accompanied by audio have become integral to our daily lives, while they often suffer from complex degradations. Most face video restoration methods neglect the intrinsic correlations between the visual and audio features, especially in mouth regions. A few audio-aided face video restoration methods have been proposed, but they only focus on compression artifact removal. In this paper, we propose a General Audio-assisted face Video restoration Network (GAVN) to address various types of streaming video distortions via identity and temporal complementary learning. Specifically, GAVN first captures inter-frame temporal features in the low-resolution space to restore frames coarsely and save computational cost. Then, GAVN extracts intra-frame identity features in the high-resolution space with the assistance of audio signals and face landmarks to restore more facial details. Finally, the reconstruction module integrates temporal features and identity features to generate high-quality face videos. Experimental results demonstrate that GAVN outperforms the existing state-of-the-art methods on face video compression artifact removal, deblurring, and super-resolution. Codes will be released upon publication.