Optimal Quantization for Distribution Synthesis

TL;DR

This paper introduces two algorithms for optimally approximating discrete probability distributions with finite precision, focusing on minimizing variational distance and informational divergence, with bounds and examples demonstrating their effectiveness.

Contribution

The paper presents novel algorithms for optimal distribution approximation using variational distance and divergence, with theoretical bounds and practical examples.

Findings

Algorithms achieve asymptotically tight bounds.

Variational distance and divergence optimal approximations can differ significantly.

Examples demonstrate the effectiveness of the proposed methods.

Abstract

Finite precision approximations of discrete probability distributions are considered, applicable for distribution synthesis, e.g., probabilistic shaping. Two algorithms are presented that find the optimal $M$-type approximation $Q$ of a distribution $P$ in terms of the variational distance $| Q-P|_1$ and the informational divergence $\mathbb{D}(Q| P)$. Bounds on the approximation errors are derived and shown to be asymptotically tight. Several examples illustrate that the variational distance optimal approximation can be quite different from the informational divergence optimal approximation.