# Statistical estimation of the Kullback-Leibler divergence

## TL;DR

This paper develops conditions under which estimators of the Kullback-Leibler divergence, based on k-nearest neighbor statistics, are asymptotically unbiased and consistent for probability measures in R^d, including Gaussian measures.

## Contribution

It introduces new asymptotic unbiasedness and consistency results for Kullback-Leibler divergence estimators using k-nearest neighbor methods, applicable to Gaussian measures.

## Key findings

- Estimates are asymptotically unbiased under wide conditions.
- Estimates are L^2-consistent for a broad class of probability measures.
- New results on Kozachenko-Leonenko entropy estimators are derived.

## Abstract

Wide conditions are provided to guarantee asymptotic unbiasedness and L^2-consistency of the introduced estimates of the Kullback-Leibler divergence for probability measures in R^d having densities w.r.t. the Lebesgue measure. These estimates are constructed by means of two independent collections of i.i.d. observations and involve the specified k-nearest neighbor statistics. In particular, the established results are valid for estimates of the Kullback-Leibler divergence between any two Gaussian measures in R^d with nondegenerate covariance matrices. As a byproduct we obtain new statements concerning the Kozachenko-Leonenko estimators of the Shannon differential entropy.