Estimating Rate-Distortion Functions Using the Energy-Based Model

TL;DR

This paper introduces an energy-based model framework that efficiently estimates high-dimensional rate-distortion functions and reconstructs optimal distributions using a single neural network and MCMC sampling.

Contribution

It presents a novel energy-based modeling approach connecting rate-distortion duality with statistical physics, enabling effective high-dimensional RD estimation.

Findings

Accurately estimates high-dimensional RD functions.

Reconstructs optimal conditional distributions effectively.

Requires only one neural network for training.

Abstract

The rate-distortion (RD) theory is one of the key concepts in information theory, providing theoretical limits for compression performance and guiding the source coding design, with both theoretical and practical significance. The Blahut-Arimoto (BA) algorithm, as a classical algorithm to compute RD functions, encounters computational challenges when applied to high-dimensional scenarios. In recent years, many neural methods have attempted to compute high-dimensional RD problems from the perspective of implicit generative models. Nevertheless, these approaches often neglect the reconstruction of the optimal conditional distribution or rely on unreasonable prior assumptions. In face of these issues, we propose an innovative energy-based modeling framework that leverages the connection between the RD dual form and the free energy in statistical physics, achieving effective reconstruction of the optimal conditional distribution.The proposed algorithm requires training only a single neural network and circumvents the challenge of computing the normalization factor in energy-based models using the Markov chain Monte Carlo (MCMC) sampling. Experimental results demonstrate the significant effectiveness of the proposed algorithm in estimating high-dimensional RD functions and reconstructing the optimal conditional distribution.