Estimating the variance of Shannon entropy

TL;DR

This paper investigates the statistical properties of an estimator for the variance of the plug-in Shannon entropy estimator in multinomial distributions, providing exact distributions and bounds to quantify uncertainty in entropy estimation.

Contribution

It introduces a novel estimator for the variance of the Shannon entropy plug-in estimator and characterizes its distribution, enhancing understanding of entropy estimation accuracy.

Findings

Exact distribution of the variance estimator derived

Upper bounds on entropy uncertainty established

Maximizing distributions identified

Abstract

The statistical analysis of data stemming from dynamical systems, including, but not limited to, time series, routinely relies on the estimation of information theoretical quantities, most notably Shannon entropy. To this purpose, possibly the most widespread tool is provided by the so-called plug-in estimator, whose statistical properties in terms of bias and variance were investigated since the first decade after the publication of Shannon's seminal works. In the case of an underlying multinomial distribution, while the bias can be evaluated by knowing support and data set size, variance is far more elusive. The aim of the present work is to investigate, in the multinomial case, the statistical properties of an estimator of a parameter that describes the variance of the plug-in estimator of Shannon entropy. We then exactly determine the probability distributions that maximize that parameter. The results presented here allow to set upper limits to the uncertainty of entropy assessments under the hypothesis of memoryless underlying stochastic processes.