To BEE or not to BEE: Estimating more than Entropy with Biased Entropy Estimators

TL;DR

This paper evaluates 18 biased entropy estimators across various measures and distributions, identifying the Chao-Shen and Chao-Wang-Jost estimators as the most accurate and efficient for software engineering applications.

Contribution

It provides a comprehensive empirical comparison of entropy estimators, highlighting the superior performance of Chao-Shen and Chao-Wang-Jost estimators across different conditions.

Findings

Chao-Shen and Chao-Wang-Jost estimators converge faster to true entropy.

These estimators outperform others in accuracy with increasing sample sizes.

Performance is consistent regardless of domain size and entropy measure.

Abstract

Entropy estimation plays a significant role in biology, economics, physics, communication engineering and other disciplines. It is increasingly used in software engineering, e.g. in software confidentiality, software testing, predictive analysis, machine learning, and software improvement. However accurate estimation is demonstrably expensive in many contexts, including software. Statisticians have consequently developed biased estimators that aim to accurately estimate entropy on the basis of a sample. In this paper we apply 18 widely employed entropy estimators to Shannon measures useful to the software engineer: entropy, mutual information and conditional mutual information. Moreover, we investigate how the estimators are affected by two main influential factors: sample size and domain size. Our experiments range over a large set of randomly generated joint probability distributions and varying sample sizes, rather than choosing just one or two well known probability distributions as in previous investigations. Our most important result is identifying that the Chao-Shen and Chao-Wang-Jost estimators stand out for consistently converging more quickly to the ground truth, regardless of domain size and regardless of the measure used. They also tend to outperform the others in terms of accuracy as sample sizes increase. This discovery enables a significant reduction in data collection effort without compromising performance.