Beyond GFVC: A Progressive Face Video Compression Framework with Adaptive Visual Tokens

TL;DR

This paper introduces PFVC, a progressive face video compression framework using adaptive visual tokens, which improves robustness, flexibility, and rate-distortion performance over existing methods and codecs.

Contribution

The novel PFVC framework employs adaptive visual tokens for progressive encoding and decoding, enhancing face video compression robustness and flexibility beyond current GFVC and VVC standards.

Findings

Outperforms latest VVC codec in rate-distortion trade-offs

Achieves more flexible and robust face video reconstruction

Demonstrates superior performance compared to state-of-the-art GFVC algorithms

Abstract

Recently, deep generative models have greatly advanced the progress of face video coding towards promising rate-distortion performance and diverse application functionalities. Beyond traditional hybrid video coding paradigms, Generative Face Video Compression (GFVC) relying on the strong capabilities of deep generative models and the philosophy of early Model-Based Coding (MBC) can facilitate the compact representation and realistic reconstruction of visual face signal, thus achieving ultra-low bitrate face video communication. However, these GFVC algorithms are sometimes faced with unstable reconstruction quality and limited bitrate ranges. To address these problems, this paper proposes a novel Progressive Face Video Compression framework, namely PFVC, that utilizes adaptive visual tokens to realize exceptional trade-offs between reconstruction robustness and bandwidth intelligence. In particular, the encoder of the proposed PFVC projects the high-dimensional face signal into adaptive visual tokens in a progressive manner, whilst the decoder can further reconstruct these adaptive visual tokens for motion estimation and signal synthesis with different granularity levels. Experimental results demonstrate that the proposed PFVC framework can achieve better coding flexibility and superior rate-distortion performance in comparison with the latest Versatile Video Coding (VVC) codec and the state-of-the-art GFVC algorithms. The project page can be found at https://github.com/Berlin0610/PFVC.