Efficient Video Diffusion with Sparse Information Transmission for Video Compression

TL;DR

This paper introduces Diff-SIT, a novel video compression method that uses sparse encoding and diffusion models to achieve high perceptual quality and temporal consistency at ultra-low bitrates.

Contribution

The paper proposes a new framework combining sparse temporal encoding and diffusion-based reconstruction to improve ultra-low bitrate video compression.

Findings

Achieves state-of-the-art perceptual quality at ultra-low bitrates.

Maintains strong temporal consistency across frames.

Reduces bitrate significantly compared to existing methods.

Abstract

Video compression aims to maximize reconstruction quality with minimal bitrates. Beyond standard distortion metrics, perceptual quality and temporal consistency are also critical. However, at ultra-low bitrates, traditional end-to-end compression models tend to produce blurry images of poor perceptual quality. Besides, existing generative compression methods often treat video frames independently and show limitations in time coherence and efficiency. To address these challenges, we propose the Efficient Video Diffusion with Sparse Information Transmission (Diff-SIT), which comprises the Sparse Temporal Encoding Module (STEM) and the One-Step Video Diffusion with Frame Type Embedder (ODFTE). The STEM sparsely encodes the original frame sequence into an information-rich intermediate sequence, achieving significant bitrate savings. Subsequently, the ODFTE processes this intermediate sequence as a whole, which exploits the temporal correlation. During this process, our proposed Frame Type Embedder (FTE) guides the diffusion model to perform adaptive reconstruction according to different frame types to optimize the overall quality. Extensive experiments on multiple datasets demonstrate that Diff-SIT establishes a new state-of-the-art in perceptual quality and temporal consistency, particularly in the challenging ultra-low-bitrate regime. Code is released at https://github.com/MingdeZhou/Diff-SIT.