Generative Models at the Frontier of Compression: A Survey on Generative Face Video Coding

TL;DR

This survey reviews the emerging field of Generative Face Video Coding (GFVC), highlighting its potential to revolutionize face video compression through deep generative models, and discusses standardization, benchmarking, and future directions.

Contribution

This paper provides the first comprehensive survey of GFVC, including a review of methods, benchmarking analysis, a new face video database, and insights into standardization and future industrial applications.

Findings

GFVC achieves ultra-low bitrate high-fidelity face video reconstruction.

Benchmarking reveals key differences among GFVC methods.

A large-scale GFVC face video database was constructed for evaluation.

Abstract

The rise of deep generative models has greatly advanced video compression, reshaping the paradigm of face video coding through their powerful capability for semantic-aware representation and lifelike synthesis. Generative Face Video Coding (GFVC) stands at the forefront of this revolution, which could characterize complex facial dynamics into compact latent codes for bitstream compactness at the encoder side and leverages powerful deep generative models to reconstruct high-fidelity face signal from the compressed latent codes at the decoder side. As such, this well-designed GFVC paradigm could enable high-fidelity face video communication at ultra-low bitrate ranges, far surpassing the capabilities of the latest Versatile Video Coding (VVC) standard. To pioneer foundational research and accelerate the evolution of GFVC, this paper presents the first comprehensive survey of GFVC technologies, systematically bridging critical gaps between theoretical innovation and industrial standardization. In particular, we first review a broad range of existing GFVC methods with different feature representations and optimization strategies, and conduct a thorough benchmarking analysis. In addition, we construct a large-scale GFVC-compressed face video database with subjective Mean Opinion Scores (MOSs) based on human perception, aiming to identify the most appropriate quality metrics tailored to GFVC. Moreover, we summarize the GFVC standardization potentials with a unified high-level syntax and develop a low-complexity GFVC system which are both expected to push forward future practical deployments and applications. Finally, we envision the potential of GFVC in industrial applications and deliberate on the current challenges and future opportunities.