Mixture-based estimation of entropy

TL;DR

This paper introduces a semi-parametric entropy estimation method using mixture models, particularly Gaussian mixtures, which improves accuracy and versatility in estimating entropy from data samples.

Contribution

The paper proposes a novel mixture-based entropy estimator that is flexible and effective, demonstrated through simulations and real data applications.

Findings

Accurate entropy estimates using Gaussian mixture models.

Versatile approach applicable to various data distributions.

Validated through simulation studies and real-world examples.

Abstract

The entropy is a measure of uncertainty that plays a central role in information theory. When the distribution of the data is unknown, an estimate of the entropy needs be obtained from the data sample itself. We propose a semi-parametric estimate, based on a mixture model approximation of the distribution of interest. The estimate can rely on any type of mixture, but we focus on Gaussian mixture model to demonstrate its accuracy and versatility. Performance of the proposed approach is assessed through a series of simulation studies. We also illustrate its use on two real-life data examples.