Importance Matching Lemma for Lossy Compression with Side Information

TL;DR

This paper introduces an importance matching lemma and an improved importance sampling-based compression scheme, enabling finite-sample evaluation and effective distributed lossy compression with side information, validated through experiments.

Contribution

The paper presents the importance matching lemma as a finite-sample counterpart to the Poisson matching lemma, and extends importance sampling methods for lossy compression with practical evaluation.

Findings

Finite-sample evaluation of compression rate achieved.

Effective distributed lossy compression demonstrated with deep learning.

Successful experiments on Gaussian sources, MNIST, and CIFAR-10.

Abstract

We propose two extensions to existing importance sampling based methods for lossy compression. First, we introduce an importance sampling based compression scheme that is a variant of ordered random coding (Theis and Ahmed, 2022) and is amenable to direct evaluation of the achievable compression rate for a finite number of samples. Our second and major contribution is the importance matching lemma, which is a finite proposal counterpart of the recently introduced Poisson matching lemma (Li and Anantharam, 2021). By integrating with deep learning, we provide a new coding scheme for distributed lossy compression with side information at the decoder. We demonstrate the effectiveness of the proposed scheme through experiments involving synthetic Gaussian sources, distributed image compression with MNIST and vertical federated learning with CIFAR-10.