Statistical estimation of the Shannon entropy

TL;DR

This paper analyzes the asymptotic properties of Kozachenko-Leonenko estimators for differential Shannon entropy, establishing their unbiasedness and consistency under certain conditions, including for Gaussian vectors.

Contribution

It provides theoretical results on the asymptotic behavior of entropy estimators, extending their validity to Gaussian vectors and involving Hardy-Littlewood maximal function conditions.

Findings

Estimates are asymptotically unbiased and consistent.

Results apply to nondegenerate Gaussian vectors.

Conditions involve Hardy-Littlewood maximal functions.

Abstract

The behavior of the Kozachenko - Leonenko estimates for the (differential) Shannon entropy is studied when the number of i.i.d. vector-valued observations tends to infinity. The asymptotic unbiasedness and L^2-consistency of the estimates are established. The conditions employed involve the analogues of the Hardy - Littlewood maximal function. It is shown that the results are valid in particular for the entropy estimation of any nondegenerate Gaussian vector.