Joint Reference Frame Synthesis and Post Filter Enhancement for Versatile Video Coding

TL;DR

This paper introduces a joint neural network-based approach for synthesizing reference frames and enhancing post-filter quality in Versatile Video Coding, significantly improving compression efficiency and artifact reduction.

Contribution

It proposes a unified neural network framework, STENet, for both reference frame synthesis and artifact suppression, integrated into VVC with reduced inference complexity.

Findings

Achieves up to 17.21% BD-rate reduction in VVC.

Effectively suppresses artifacts and distortions in reconstructed frames.

Demonstrates the benefits of joint RFS and PFE in video coding efficiency.

Abstract

This paper presents the joint reference frame synthesis (RFS) and post-processing filter enhancement (PFE) for Versatile Video Coding (VVC), aiming to explore the combination of different neural network-based video coding (NNVC) tools to better utilize the hierarchical bi-directional coding structure of VVC. Both RFS and PFE utilize the Space-Time Enhancement Network (STENet), which receives two input frames with artifacts and produces two enhanced frames with suppressed artifacts, along with an intermediate synthesized frame. STENet comprises two pipelines, the synthesis pipeline and the enhancement pipeline, tailored for different purposes. During RFS, two reconstructed frames are sent into STENet's synthesis pipeline to synthesize a virtual reference frame, similar to the current to-be-coded frame. The synthesized frame serves as an additional reference frame inserted into the reference picture list (RPL). During PFE, two reconstructed frames are fed into STENet's enhancement pipeline to alleviate their artifacts and distortions, resulting in enhanced frames with reduced artifacts and distortions. To reduce inference complexity, we propose joint inference of RFS and PFE (JISE), achieved through a single execution of STENet. Integrated into the VVC reference software VTM-15.0, RFS, PFE, and JISE are coordinated within a novel Space-Time Enhancement Window (STEW) under Random Access (RA) configuration. The proposed method could achieve -7.34%/-17.21%/-16.65% PSNR-based BD-rate on average for three components under RA configuration.