Sandwiched Compression: Repurposing Standard Codecs with Neural Network Wrappers

TL;DR

This paper introduces a novel approach called sandwiching, where neural networks are used to enhance standard codecs by joint training with differentiable proxies, leading to significant improvements in rate-distortion performance across various content types and distortion measures.

Contribution

The paper presents a new sandwich architecture that adapts standard codecs with neural network wrappers, improving performance and flexibility beyond traditional methods.

Findings

Up to 9 dB gain in quality

30% bitrate reduction

Effective across diverse content and distortion measures

Abstract

We propose sandwiching standard image and video codecs between pre- and post-processing neural networks. The networks are jointly trained through a differentiable codec proxy to minimize a given rate-distortion loss. This sandwich architecture not only improves the standard codec's performance on its intended content, but more importantly, adapts the codec to other types of image/video content and to other distortion measures. The sandwich learns to transmit ``neural code images'' that optimize and improve overall rate-distortion performance, with the improvements becoming significant especially when the overall problem is well outside of the scope of the codec's design. We apply the sandwich architecture to standard codecs with mismatched sources transporting different numbers of channels, higher resolution, higher dynamic range, computer graphics, and with perceptual distortion measures. The results demonstrate substantial improvements (up to 9 dB gains or up to 30\% bitrate reductions) compared to alternative adaptations. We establish optimality properties for sandwiched compression and design differentiable codec proxies approximating current standard codecs. We further analyze model complexity, visual quality under perceptual metrics, as well as sandwich configurations that offer interesting potentials in video compression and streaming.