Learned Hierarchical B-frame Coding with Adaptive Feature Modulation for YUV 4:2:0 Content

TL;DR

This paper presents a novel learned hierarchical B-frame video coding scheme for YUV 4:2:0 content that uses adaptive feature modulation and content-aware variable-rate coding, outperforming traditional codecs.

Contribution

The work introduces a new learned B-frame coding method with adaptive feature modulation and content-aware variable-rate coding specifically for YUV 4:2:0 video content, addressing gaps in prior neural codecs.

Findings

Outperforms x265 in PSNR-YUV on standard datasets

Effective content-adaptive variable-rate coding achieved

Utilizes conditional inter-frame codecs for Y and UV components

Abstract

This paper introduces a learned hierarchical B-frame coding scheme in response to the Grand Challenge on Neural Network-based Video Coding at ISCAS 2023. We address specifically three issues, including (1) B-frame coding, (2) YUV 4:2:0 coding, and (3) content-adaptive variable-rate coding with only one single model. Most learned video codecs operate internally in the RGB domain for P-frame coding. B-frame coding for YUV 4:2:0 content is largely under-explored. In addition, while there have been prior works on variable-rate coding with conditional convolution, most of them fail to consider the content information. We build our scheme on conditional augmented normalized flows (CANF). It features conditional motion and inter-frame codecs for efficient B-frame coding. To cope with YUV 4:2:0 content, two conditional inter-frame codecs are used to process the Y and UV components separately, with the coding of the UV components conditioned additionally on the Y component. Moreover, we introduce adaptive feature modulation in every convolutional layer, taking into account both the content information and the coding levels of B-frames to achieve content-adaptive variable-rate coding. Experimental results show that our model outperforms x265 and the winner of last year's challenge on commonly used datasets in terms of PSNR-YUV.