Characterisations of Kullback--Leibler approximation by finite Gaussian mixtures

TL;DR

This paper characterizes when Gaussian mixture models can approximate a target density in KL divergence, linking it to properties like finite second moments and constructing specific GMMs.

Contribution

It provides necessary and sufficient conditions for KL approximation by finite GMMs, including new classes of target densities and counterexamples.

Findings

Finite second moment is necessary for KL approximation by GMMs.

Constructive conditions for GMM approximation involve pointwise likelihood ratio convergence.

Counterexamples show the limits of classes of target densities for KL approximation.

Abstract

We study the Kullback--Leibler (KL) divergence approximation theory of Gaussian mixture models (GMMs) by isolating an abstract mechanism behind several necessary-and-sufficient statements. The necessity direction is universal: if a density is approximable in KL divergence by finite GMMs, then it must have finite second moment. The sufficient direction is reduced to the construction of approximating GMMs whose likelihood ratios converge pointwise and whose finite log-ratios form a uniformly integrable family. We verify this mechanism on a finite log-moment class of continuous strictly positive target densities, from which bounded, $\mathcal L^p$ $(p>1)$, and Orlicz-dominated subfamilies follow immediately. We also show that a countable-scale support-aware target density class, which allows zero density regions, satisfies the same equivalence. Finally, we give counterexamples showing that the countable-scale class strictly extends the fixed-scale class, that the finite log-moment and countable-scale support-aware classes do not contain one another, and that their union is not exhaustive.