Analysis of KNN Information Estimators for Smooth Distributions

TL;DR

This paper analyzes the convergence rate of the KSG mutual information estimator for smooth distributions, extending understanding beyond previously studied bounded densities to include unbounded and near-zero densities.

Contribution

It provides a new convergence analysis of the KSG estimator for a broader class of smooth distributions, including unbounded support and densities approaching zero.

Findings

Convergence rate of KSG estimator for smooth distributions is characterized.

Analysis includes distributions with unbounded support and densities near zero.

Provides convergence analysis for KL entropy estimator for broad distribution classes.

Abstract

KSG mutual information estimator, which is based on the distances of each sample to its k-th nearest neighbor, is widely used to estimate mutual information between two continuous random variables. Existing work has analyzed the convergence rate of this estimator for random variables whose densities are bounded away from zero in its support. In practice, however, KSG estimator also performs well for a much broader class of distributions, including not only those with bounded support and densities bounded away from zero, but also those with bounded support but densities approaching zero, and those with unbounded support. In this paper, we analyze the convergence rate of the error of KSG estimator for smooth distributions, whose support of density can be both bounded and unbounded. As KSG mutual information estimator can be viewed as an adaptive recombination of KL entropy estimators, in our analysis, we also provide convergence analysis of KL entropy estimator for a broad class of distributions.