A note on entropy estimation

TL;DR

This paper compares different entropy estimators, proves their relationships, and demonstrates that some estimators outperform others in small sample, large event space scenarios through theoretical proofs and numerical simulations.

Contribution

It establishes the equivalence of two entropy estimators and compares their biases and errors, highlighting the advantages of estimator H_2 in specific regimes.

Findings

H_z is equivalent to H_1, correcting previous assumptions.

H_1 has a smaller bias than the likelihood estimator.

H_2 exhibits significantly lower statistical error in small sample, large event space regimes.

Abstract

We compare an entropy estimator $H_z$ recently discussed in [10] with two estimators $H_1$ and $H_2$ introduced in [6][7]. We prove the identity $H_z \equiv H_1$, which has not been taken into account in [10]. Then, we prove that the statistical bias of $H_1$ is less than the bias of the ordinary likelihood estimator of entropy. Finally, by numerical simulation we verify that for the most interesting regime of small sample estimation and large event spaces, the estimator $H_2$ has a significant smaller statistical error than $H_z$.