Diffusion-based Perceptual Neural Video Compression with Temporal Diffusion Information Reuse

TL;DR

This paper introduces DiffVC, a diffusion-based neural video compression framework that leverages temporal information reuse and quantization parameter prompting to improve efficiency and quality across variable bitrates.

Contribution

The paper presents a novel diffusion-based video compression method with a temporal diffusion information reuse strategy and a quantization parameter prompting mechanism, enhancing efficiency and adaptability.

Findings

Achieves high perceptual quality and visual fidelity in video compression.

Significantly improves inference efficiency with minimal performance loss.

Effective across various bitrates with robust quality.

Abstract

Recently, foundational diffusion models have attracted considerable attention in image compression tasks, whereas their application to video compression remains largely unexplored. In this article, we introduce DiffVC, a diffusion-based perceptual neural video compression framework that effectively integrates foundational diffusion model with the video conditional coding paradigm. This framework uses temporal context from previously decoded frame and the reconstructed latent representation of the current frame to guide the diffusion model in generating high-quality results. To accelerate the iterative inference process of diffusion model, we propose the Temporal Diffusion Information Reuse (TDIR) strategy, which significantly enhances inference efficiency with minimal performance loss by reusing the diffusion information from previous frames. Additionally, to address the challenges posed by distortion differences across various bitrates, we propose the Quantization Parameter-based Prompting (QPP) mechanism, which utilizes quantization parameters as prompts fed into the foundational diffusion model to explicitly modulate intermediate features, thereby enabling a robust variable bitrate diffusion-based neural compression framework. Experimental results demonstrate that our proposed solution delivers excellent performance in both perception metrics and visual quality.