On the consistency of the Kozachenko-Leonenko entropy estimate

TL;DR

This paper analyzes the conditions under which the Kozachenko-Leonenko nearest neighbor estimator for differential entropy is consistent, showing it converges under minimal assumptions and almost surely with compact support.

Contribution

It establishes the necessary and sufficient conditions for the estimator's consistency without smoothness assumptions, including almost sure convergence for compactly supported distributions.

Findings

Estimator is consistent if and only if expected log norm is finite.

Almost sure convergence occurs for distributions with compact support.

Provides minimal conditions for the estimator's reliability.

Abstract

We revisit the problem of the estimation of the differential entropy $H(f)$ of a random vector $X$ in $R^d$ with density $f$, assuming that $H(f)$ exists and is finite. In this note, we study the consistency of the popular nearest neighbor estimate $H_n$ of Kozachenko and Leonenko. Without any smoothness condition we show that the estimate is consistent ($E\{|H_n - H(f)|\} \to 0$ as $n \to \infty$) if and only if $\mathbb{E} \{ \log ( \| X \| + 1 )\} < \infty$. Furthermore, if $X$ has compact support, then $H_n \to H(f)$ almost surely.