Neural B-frame Video Compression with Bi-directional Reference Harmonization

TL;DR

This paper introduces BRHVC, a novel neural B-frame video compression method that effectively harmonizes bi-directional references using motion convergence and contextual fusion, outperforming existing neural and traditional codecs.

Contribution

The paper proposes BRHVC, a new neural B-frame compression approach with bi-directional reference harmonization, including BMC and BCF modules, improving motion compensation and context modeling.

Findings

BRHVC outperforms previous neural video compression methods.

BRHVC surpasses traditional VTM-RA in HEVC datasets.

The source code is publicly available.

Abstract

Neural video compression (NVC) has made significant progress in recent years, while neural B-frame video compression (NBVC) remains underexplored compared to P-frame compression. NBVC can adopt bi-directional reference frames for better compression performance. However, NBVC's hierarchical coding may complicate continuous temporal prediction, especially at some hierarchical levels with a large frame span, which could cause the contribution of the two reference frames to be unbalanced. To optimize reference information utilization, we propose a novel NBVC method, termed Bi-directional Reference Harmonization Video Compression (BRHVC), with the proposed Bi-directional Motion Converge (BMC) and Bi-directional Contextual Fusion (BCF). BMC converges multiple optical flows in motion compression, leading to more accurate motion compensation on a larger scale. Then BCF explicitly models the weights of reference contexts under the guidance of motion compensation accuracy. With more efficient motions and contexts, BRHVC can effectively harmonize bi-directional references. Experimental results indicate that our BRHVC outperforms previous state-of-the-art NVC methods, even surpassing the traditional coding, VTM-RA (under random access configuration), on the HEVC datasets. The source code is released at https://github.com/kwai/NVC.