A class of R\'{e}nyi information estimators for multidimensional densities

TL;DR

This paper introduces a new class of estimators for Rényi and Tsallis entropies in multidimensional spaces, utilizing nearest-neighbor distances, and demonstrates their consistency and applicability to distribution comparison.

Contribution

It presents a novel, consistent method for estimating Rényi and Tsallis entropies in multiple dimensions using nearest-neighbor distances, extending to distribution distance estimation.

Findings

Estimators are consistent for any order q, including Shannon entropy.

Method requires minimal assumptions on the distribution f.

Extension to statistical distance estimation between distributions.

Abstract

A class of estimators of the R\'{e}nyi and Tsallis entropies of an unknown distribution $f$ in $\mathbb{R}^m$ is presented. These estimators are based on the $k$th nearest-neighbor distances computed from a sample of $N$ i.i.d. vectors with distribution $f$. We show that entropies of any order $q$, including Shannon's entropy, can be estimated consistently with minimal assumptions on $f$. Moreover, we show that it is straightforward to extend the nearest-neighbor method to estimate the statistical distance between two distributions using one i.i.d. sample from each. (Wit Correction.)