Greedy Algorithms for Optimal Distribution Approximation

TL;DR

This paper investigates greedy algorithms for approximating a discrete probability distribution with an M-type distribution, minimizing divergence or variational distance, and provides theoretical bounds and properties of optimal solutions.

Contribution

It introduces a unified greedy algorithm framework for optimal distribution approximation minimizing divergence or variational distance, with proven properties and asymptotic bounds.

Findings

Optimal approximation properties are derived.

Bounds on approximation error are asymptotically tight.

A single greedy algorithm can find optimal solutions for different divergence measures.

Abstract

The approximation of a discrete probability distribution $\mathbf{t}$ by an $M$-type distribution $\mathbf{p}$ is considered. The approximation error is measured by the informational divergence $\mathbb{D}(\mathbf{t}\Vert\mathbf{p})$, which is an appropriate measure, e.g., in the context of data compression. Properties of the optimal approximation are derived and bounds on the approximation error are presented, which are asymptotically tight. It is shown that $M$-type approximations that minimize either $\mathbb{D}(\mathbf{t}\Vert\mathbf{p})$, or $\mathbb{D}(\mathbf{p}\Vert\mathbf{t})$, or the variational distance $\Vert\mathbf{p}-\mathbf{t}\Vert_1$ can all be found by using specific instances of the same general greedy algorithm.