Finite-Sample Analysis of Fixed-k Nearest Neighbor Density Functional Estimators

TL;DR

This paper introduces a finite-sample analysis for fixed-k nearest neighbor estimators of density functionals, demonstrating improved efficiency and convergence rates over traditional methods that rely on increasing k.

Contribution

It provides the first finite-sample guarantees for a unified framework of fixed-k k-nearest neighbor estimators for density functionals, including bias correction techniques.

Findings

Establishes finite-sample bounds for the estimators.

Shows faster convergence rates compared to traditional methods.

Unifies multiple existing estimators under a common framework.

Abstract

We provide finite-sample analysis of a general framework for using k-nearest neighbor statistics to estimate functionals of a nonparametric continuous probability density, including entropies and divergences. Rather than plugging a consistent density estimate (which requires $k \to \infty$ as the sample size $n \to \infty$) into the functional of interest, the estimators we consider fix k and perform a bias correction. This is more efficient computationally, and, as we show in certain cases, statistically, leading to faster convergence rates. Our framework unifies several previous estimators, for most of which ours are the first finite sample guarantees.