On the Entropy of General Mixture Distributions

TL;DR

This paper introduces a deterministic, closed-form method for bounding and approximating the differential entropy of mixture distributions, leveraging an information-theoretic channel perspective and pairwise component overlaps.

Contribution

It develops a novel toolkit for accurately estimating mixture entropy directly from component parameters, including explicit bounds and approximations with bias correction.

Findings

Validated bounds and approximations across various mixture types.

Effective in high-dimensional and correlated component scenarios.

Provides closed-form expressions for common mixture families.

Abstract

Mixture distributions are a workhorse model for multimodal data in information theory, signal processing, and machine learning. Yet even when each component density is simple, the differential entropy of the mixture is notoriously hard to compute because the mixture couples a logarithm with a sum. This paper develops a deterministic, closed-form toolkit for bounding and accurately approximating mixture entropy directly from component parameters. Our starting point is an information-theoretic channel viewpoint: the latent mixture label plays the role of an input, and the observation is the output. This viewpoint separates mixture entropy into an average within-component uncertainty plus an overlap term that quantifies how much the observation reveals about the hidden label. We then bound and approximate this overlap term using pairwise overlap integrals between component densities, yielding explicit expressions whenever these overlaps admit a closed form. A simple, family-dependent offset corrects the systematic bias of the Jensen overlap bound and is calibrated to be exact in the two limiting regimes of complete overlap and near-perfect separation. A final clipping step guarantees that the estimate always respects universal information-theoretic bounds. Closed-form specializations are provided for Gaussian, factorized Laplacian, uniform, and hybrid mixtures, and numerical experiments validate the resulting bounds and approximations across separation, dimension, number of components, and correlated covariances.