Guaranteed bounds on the Kullback-Leibler divergence of univariate mixtures using piecewise log-sum-exp inequalities

TL;DR

This paper introduces a fast, generic method to compute tight bounds on the Kullback-Leibler divergence between univariate mixture models, addressing the lack of closed-form solutions in a computationally efficient way.

Contribution

It presents a novel algorithmic approach to derive closed-form bounds on divergence measures for various mixture models, improving efficiency over existing methods.

Findings

The method provides tight lower and upper bounds for univariate mixtures.

It is applicable to exponential, Gaussian, Rayleigh, and Gamma mixture models.

Experiments demonstrate improved computational efficiency and accuracy.

Abstract

Information-theoretic measures such as the entropy, cross-entropy and the Kullback-Leibler divergence between two mixture models is a core primitive in many signal processing tasks. Since the Kullback-Leibler divergence of mixtures provably does not admit a closed-form formula, it is in practice either estimated using costly Monte-Carlo stochastic integration, approximated, or bounded using various techniques. We present a fast and generic method that builds algorithmically closed-form lower and upper bounds on the entropy, the cross-entropy and the Kullback-Leibler divergence of mixtures. We illustrate the versatile method by reporting on our experiments for approximating the Kullback-Leibler divergence between univariate exponential mixtures, Gaussian mixtures, Rayleigh mixtures, and Gamma mixtures.