Neural Estimation of the Rate-Distortion Function With Applications to Operational Source Coding

TL;DR

This paper introduces NERD, a neural network-based method to estimate the rate-distortion function for real-world data, enabling comparison of neural compressors with theoretical limits and facilitating practical lossy compression schemes.

Contribution

The paper presents NERD, a novel neural network approach to accurately estimate the rate-distortion function for high-dimensional data, overcoming computational challenges of traditional methods.

Findings

NERD accurately estimates the rate-distortion function on image datasets.

DNN compressors operate within several bits of the theoretical rate-distortion limit.

NERD enables construction of operational lossy compression schemes with performance guarantees.

Abstract

A fundamental question in designing lossy data compression schemes is how well one can do in comparison with the rate-distortion function, which describes the known theoretical limits of lossy compression. Motivated by the empirical success of deep neural network (DNN) compressors on large, real-world data, we investigate methods to estimate the rate-distortion function on such data, which would allow comparison of DNN compressors with optimality. While one could use the empirical distribution of the data and apply the Blahut-Arimoto algorithm, this approach presents several computational challenges and inaccuracies when the datasets are large and high-dimensional, such as the case of modern image datasets. Instead, we re-formulate the rate-distortion objective, and solve the resulting functional optimization problem using neural networks. We apply the resulting rate-distortion estimator, called NERD, on popular image datasets, and provide evidence that NERD can accurately estimate the rate-distortion function. Using our estimate, we show that the rate-distortion achievable by DNN compressors are within several bits of the rate-distortion function for real-world datasets. Additionally, NERD provides access to the rate-distortion achieving channel, as well as samples from its output marginal. Therefore, using recent results in reverse channel coding, we describe how NERD can be used to construct an operational one-shot lossy compression scheme with guarantees on the achievable rate and distortion. Experimental results demonstrate competitive performance with DNN compressors.