Controllable Generative Video Compression

TL;DR

This paper introduces CGVC, a new controllable generative video compression method that balances perceptual quality and signal fidelity by using keyframes and control priors.

Contribution

The paper proposes a novel CGVC paradigm that uses structural and dense control priors for faithful, controllable non-keyframe video reconstruction.

Findings

CGVC outperforms previous methods in perceptual quality.

CGVC achieves higher signal fidelity in reconstructed videos.

Color-distance-guided keyframe selection improves color accuracy.

Abstract

Perceptual video compression adopts generative video modeling to improve perceptual realism but frequently sacrifices signal fidelity, diverging from the goal of video compression to faithfully reproduce visual signal. To alleviate the dilemma between perception and fidelity, in this paper we propose Controllable Generative Video Compression (CGVC) paradigm to faithfully generate details guided by multiple visual conditions. Under the paradigm, representative keyframes of the scene are coded and used to provide structural priors for non-keyframe generation. Dense per-frame control prior is additionally coded to better preserve finer structure and semantics of each non-keyframe. Guided by these priors, non-keyframes are reconstructed by controllable video generation model with temporal and content consistency. Furthermore, to accurately recover color information of the video, we develop a color-distance-guided keyframe selection algorithm to adaptively choose keyframes. Experimental results show CGVC outperforms previous perceptual video compression method in terms of both signal fidelity and perceptual quality.