Single-step Diffusion-based Video Coding with Semantic-Temporal Guidance

TL;DR

S2VC introduces a single-step diffusion-based video codec that achieves high perceptual quality and significant bitrate savings by using semantic and temporal guidance, reducing complexity compared to existing methods.

Contribution

The paper presents S2VC, a novel single-step diffusion video codec with semantic and temporal guidance, enabling efficient, realistic low-bitrate video reconstruction.

Findings

52.73% bitrate saving over prior methods

State-of-the-art perceptual quality at low bitrates

Effective semantic and temporal conditioning improves realism

Abstract

While traditional and neural video codecs (NVCs) have achieved remarkable rate-distortion performance, improving perceptual quality at low bitrates remains challenging. Some NVCs incorporate perceptual or adversarial objectives but still suffer from artifacts due to limited generation capacity, whereas others leverage pretrained diffusion models to improve quality at the cost of heavy sampling complexity. To overcome these challenges, we propose S2VC, a Single-Step diffusion based Video Codec that integrates a conditional coding framework with an efficient single-step diffusion generator, enabling realistic reconstruction at low bitrates with reduced sampling cost. Recognizing the importance of semantic conditioning in single-step diffusion, we introduce Contextual Semantic Guidance to extract frame-adaptive semantics from buffered features. It replaces text captions with efficient, fine-grained conditioning, thereby improving generation realism. In addition, Temporal Consistency Guidance is incorporated into the diffusion U-Net to enforce temporal coherence across frames and ensure stable generation. Extensive experiments show that S2VC delivers state-of-the-art perceptual quality with an average 52.73% bitrate saving over prior perceptual methods, underscoring the promise of single-step diffusion for efficient, high-quality video compression.