Optimally approximating exponential families

TL;DR

This paper investigates how well finite exponential families can approximate arbitrary probability distributions, focusing on minimal divergence bounds and optimal partition-based families, especially those with divergence equal to log(2).

Contribution

It characterizes the optimal exponential families for approximating distributions within a specified divergence bound, emphasizing partition-based constructions and the special case of divergence log(2).

Findings

Partition exponential families can achieve minimal divergence bounds.

Exponential families with divergence log(2) are thoroughly analyzed.

For divergence less than log(2), the family includes all full-support distributions.

Abstract

This article studies exponential families $\mathcal{E}$ on finite sets such that the information divergence $D(P\|\mathcal{E})$ of an arbitrary probability distribution from $\mathcal{E}$ is bounded by some constant $D>0$. A particular class of low-dimensional exponential families that have low values of $D$ can be obtained from partitions of the state space. The main results concern optimality properties of these partition exponential families. Exponential families where $D=\log(2)$ are studied in detail. This case is special, because if $D<\log(2)$, then $\mathcal{E}$ contains all probability measures with full support.