Sandwiched Video Compression: Efficiently Extending the Reach of Standard Codecs with Neural Wrappers

TL;DR

This paper introduces a neural network-based framework that wraps around standard video codecs to significantly improve compression efficiency and quality, especially in high-resolution scenarios, by jointly training neural pre- and post-processors.

Contribution

It presents a novel differentiable approximation of video codec components enabling end-to-end training of neural wrappers around standard codecs, leading to substantial compression gains.

Findings

6.5 dB improvement over standard HEVC in high-resolution video transport

30% better rate-distortion performance measured by LPIPS

Lightweight neural wrappers achieve near-optimal results

Abstract

We propose sandwiched video compression -- a video compression system that wraps neural networks around a standard video codec. The sandwich framework consists of a neural pre- and post-processor with a standard video codec between them. The networks are trained jointly to optimize a rate-distortion loss function with the goal of significantly improving over the standard codec in various compression scenarios. End-to-end training in this setting requires a differentiable proxy for the standard video codec, which incorporates temporal processing with motion compensation, inter/intra mode decisions, and in-loop filtering. We propose differentiable approximations to key video codec components and demonstrate that, in addition to providing meaningful compression improvements over the standard codec, the neural codes of the sandwich lead to significantly better rate-distortion performance in two important scenarios.When transporting high-resolution video via low-resolution HEVC, the sandwich system obtains 6.5 dB improvements over standard HEVC. More importantly, using the well-known perceptual similarity metric, LPIPS, we observe 30% improvements in rate at the same quality over HEVC. Last but not least, we show that pre- and post-processors formed by very modestly-parameterized, light-weight networks can closely approximate these results.