Pleno-Generation: A Scalable Generative Face Video Compression Framework with Bandwidth Intelligence

TL;DR

The paper introduces Pleno-Generation, a scalable face video compression framework that enhances fidelity and flexibility across a wide bitrate range by leveraging bandwidth intelligence and layered reconstruction.

Contribution

It presents a novel scalable generative face video compression framework that improves robustness and fidelity over existing methods by using layered representation and bandwidth-aware coding.

Findings

Facilitates high-fidelity face video reconstruction across a wide bitrate range.

Achieves superior rate-distortion performance compared to existing GFVC algorithms.

Attains competitive BDR savings relative to VVC codec.

Abstract

Generative model based compact video compression is typically operated within a relative narrow range of bitrates, and often with an emphasis on ultra-low rate applications. There has been an increasing consensus in the video communication industry that full bitrate coverage should be enabled by generative coding. However, this is an extremely difficult task, largely because generation and compression, although related, have distinct goals and trade-offs. The proposed Pleno-Generation (PGen) framework distinguishes itself through its exceptional capabilities in ensuring the robustness of video coding by utilizing a wider range of bandwidth for generation via bandwidth intelligence. In particular, we initiate our research of PGen with face video coding, and PGen offers a paradigm shift that prioritizes high-fidelity reconstruction over pursuing compact bitstream. The novel PGen framework leverages scalable representation and layered reconstruction for Generative Face Video Compression (GFVC), in an attempt to imbue the bitstream with intelligence in different granularity. Experimental results illustrate that the proposed PGen framework can facilitate existing GFVC algorithms to better deliver high-fidelity and faithful face videos. In addition, the proposed framework can allow a greater space of flexibility for coding applications and show superior RD performance with a much wider bitrate range in terms of various quality evaluations. Moreover, in comparison with the latest Versatile Video Coding (VVC) codec, the proposed scheme achieves competitive Bj{\o}ntegaard-delta-rate savings for perceptual-level evaluations.