Fine-Grained Motion Compression and Selective Temporal Fusion for Neural B-Frame Video Coding

TL;DR

This paper introduces novel motion compression and temporal fusion techniques for neural B-frame video coding, significantly improving compression efficiency and outperforming existing codecs.

Contribution

It presents a fine-grained motion auto-encoder with adaptive quantization and an interactive entropy model, along with a selective temporal fusion method with implicit alignment.

Findings

Achieves approximately 10% BD-rate reduction over state-of-the-art neural B-frame codec.

Delivers comparable or better compression than H.266/VVC under random-access.

Demonstrates effectiveness through extensive experiments.

Abstract

With the remarkable progress in neural P-frame video coding, neural B-frame coding has recently emerged as a critical research direction. However, most existing neural B-frame codecs directly adopt P-frame coding tools without adequately addressing the unique challenges of B-frame compression, leading to suboptimal performance. To bridge this gap, we propose novel enhancements for motion compression and temporal fusion for neural B-frame coding. First, we design a fine-grained motion compression method. This method incorporates an interactive dual-branch motion auto-encoder with per-branch adaptive quantization steps, which enables fine-grained compression of bi-directional motion vectors while accommodating their asymmetric bitrate allocation and reconstruction quality requirements. Furthermore, this method involves an interactive motion entropy model that exploits correlations between bi-directional motion latent representations by interactively leveraging partitioned latent segments as directional priors. Second, we propose a selective temporal fusion method that predicts bi-directional fusion weights to achieve discriminative utilization of bi-directional multi-scale temporal contexts with varying qualities. Additionally, this method introduces a hyperprior-based implicit alignment mechanism for contextual entropy modeling. By treating the hyperprior as a surrogate for the contextual latent representation, this mechanism implicitly mitigates the misalignment in the fused bi-directional temporal priors. Extensive experiments demonstrate that our proposed codec achieves an average BD-rate reduction of approximately 10% compared to the state-of-the-art neural B-frame codec, DCVC-B, and delivers comparable or even superior compression performance to the H.266/VVC reference software under random-access configurations.