Nearest-neighbor Entropy Estimators with Weak Metrics

TL;DR

This paper introduces weak metrics for nonparametric entropy estimation, proposing a new nearest-neighbor estimator with bias reduction and variance close to the theoretical lower bound, improving accuracy for stationary ergodic processes.

Contribution

The paper develops a novel class of weak metrics and constructs a new nearest-neighbor entropy estimator with bias optimization and near-optimal variance bounds.

Findings

Estimator's bias can be minimized using a tunable parameter.

Variance of the estimator is close to the Cramer-Rao lower bound.

The approach improves entropy estimation accuracy for stationary ergodic processes.

Abstract

A problem of improving the accuracy of nonparametric entropy estimation for a stationary ergodic process is considered. New weak metrics are introduced and relations between metrics, measures, and entropy are discussed. Based on weak metrics, a new nearest-neighbor entropy estimator is constructed and has a parameter with which the estimator is optimized to reduce its bias. It is shown that estimator's variance is upper-bounded by a nearly optimal Cramer-Rao lower bound.