Estimation of R\'enyi Entropy and Mutual Information Based on Generalized Nearest-Neighbor Graphs

TL;DR

This paper introduces simple, efficient nonparametric estimators for Re9nyi entropy and mutual information using generalized nearest-neighbor graphs, proving their consistency and demonstrating practical effectiveness in independent subspace analysis.

Contribution

The paper develops the first consistent estimators for Re9nyi entropy and mutual information based on generalized nearest-neighbor graphs, with proven convergence properties.

Findings

Estimators are computationally efficient and simple to implement.

Proven almost sure consistency of the estimators.

Effective in independent subspace analysis experiments.

Abstract

We present simple and computationally efficient nonparametric estimators of R\'enyi entropy and mutual information based on an i.i.d. sample drawn from an unknown, absolutely continuous distribution over $\R^d$. The estimators are calculated as the sum of $p$-th powers of the Euclidean lengths of the edges of the `generalized nearest-neighbor' graph of the sample and the empirical copula of the sample respectively. For the first time, we prove the almost sure consistency of these estimators and upper bounds on their rates of convergence, the latter of which under the assumption that the density underlying the sample is Lipschitz continuous. Experiments demonstrate their usefulness in independent subspace analysis.