Standard compliant video coding using low complexity, switchable neural wrappers

TL;DR

This paper introduces a standard-compatible neural wrapper framework for video coding that improves rate-distortion performance with low complexity, enabling better integration of neural processing into existing codecs.

Contribution

It proposes a novel neural wrapper architecture with optimized pre- and post-processors, achieving significant BD-Rate reductions while maintaining low decoding complexity.

Findings

Achieves 9.3% BD-Rate reduction over VVC on UVG dataset.

Achieves 6.4% BD-Rate reduction on AOM CTC Class A1.

Post-processor complexity is only 516 MACs per pixel.

Abstract

The proliferation of high resolution videos posts great storage and bandwidth pressure on cloud video services, driving the development of next-generation video codecs. Despite great progress made in neural video coding, existing approaches are still far from economical deployment considering the complexity and rate-distortion performance tradeoff. To clear the roadblocks for neural video coding, in this paper we propose a new framework featuring standard compatibility, high performance, and low decoding complexity. We employ a set of jointly optimized neural pre- and post-processors, wrapping a standard video codec, to encode videos at different resolutions. The rate-distorion optimal downsampling ratio is signaled to the decoder at the per-sequence level for each target rate. We design a low complexity neural post-processor architecture that can handle different upsampling ratios. The change of resolution exploits the spatial redundancy in high-resolution videos, while the neural wrapper further achieves rate-distortion performance improvement through end-to-end optimization with a codec proxy. Our light-weight post-processor architecture has a complexity of 516 MACs / pixel, and achieves 9.3% BD-Rate reduction over VVC on the UVG dataset, and 6.4% on AOM CTC Class A1. Our approach has the potential to further advance the performance of the latest video coding standards using neural processing with minimal added complexity.