Estimation of Shannon differential entropy: An extensive comparative review

TL;DR

This paper extensively compares 27 estimators of Shannon differential entropy across various sample sizes, dimensions, and distribution types, highlighting their relative performances and proposing optimal estimators for different scenarios.

Contribution

It provides a comprehensive empirical comparison of three classes of entropy estimators and introduces a new optimal window size class for improved estimation accuracy.

Findings

Spacings estimators perform better in univariate cases.

kNN estimators are more reliable across all dimensions.

Recommended estimators vary by distribution type and dimensionality.

Abstract

In this research work, a total of 45 different estimators of the Shannon differential entropy were reviewed. The estimators were mainly based on three classes, namely: window size spacings, kernel density estimation (KDE) and k-nearest neighbour (kNN) estimation. A total of 16, 5 and 6 estimators were selected from each of the classes, respectively, for comparison. The performances of the 27 selected estimators, in terms of their bias values and root mean squared errors (RMSEs) as well as their asymptotic behaviours, were compared through extensive Monte Carlo simulations. The empirical comparisons were carried out at different sample sizes of 10, 50, and 100 and different variable dimensions of 1, 2, 3, and 5, for three groups of continuous distributions according to their symmetry and support. The results showed that the spacings based estimators generally performed better than the estimators from the other two classes at univariate level, but suffered from non existence at multivariate level. The kNN based estimators were generally inferior to the estimators from the other two classes considered but showed an advantage of existence for all dimensions. Also, a new class of optimal window size was obtained and sets of estimators were recommended for different groups of distributions at different variable dimensions. Finally, the asymptotic biases, variances and distributions of the 'best estimators' were considered.