Geometric k-nearest neighbor estimation of entropy and mutual information

TL;DR

This paper introduces geometric k-nearest neighbor estimators that adapt local volume elements to better capture the local geometry of data, improving the estimation of entropy and mutual information especially in complex dynamical systems.

Contribution

The paper proposes a new class of geometric knn estimators using elliptical volume elements to enhance local geometry modeling in entropy and mutual information estimation.

Findings

g-knn estimators outperform traditional methods in complex local geometries

Local geometry significantly impacts knn-based estimators

g-knn is effective with limited data in large systems

Abstract

Nonparametric estimation of mutual information is used in a wide range of scientific problems to quantify dependence between variables. The k-nearest neighbor (knn) methods are consistent, and therefore expected to work well for large sample size. These methods use geometrically regular local volume elements. This practice allows maximum localization of the volume elements, but can also induce a bias due to a poor description of the local geometry of the underlying probability measure. We introduce a new class of knn estimators that we call geometric knn estimators (g-knn), which use more complex local volume elements to better model the local geometry of the probability measures. As an example of this class of estimators, we develop a g-knn estimator of entropy and mutual information based on elliptical volume elements, capturing the local stretching and compression common to a wide range of dynamical systems attractors. A series of numerical examples in which the thickness of the underlying distribution and the sample sizes are varied suggest that local geometry is a source of problems for knn methods such as the Kraskov-St\"{o}gbauer-Grassberger (KSG) estimator when local geometric effects cannot be removed by global preprocessing of the data. The g-knn method performs well despite the manipulation of the local geometry. In addition, the examples suggest that the g-knn estimators can be of particular relevance to applications in which the system is large, but data size is limited.