A New Estimator of Kullback--Leibler Divergence via Shannon Entropy

TL;DR

This paper introduces a novel estimator for Kullback-Leibler divergence based on Shannon entropy and k-nearest neighbor methods, providing an effective goodness-of-fit test for multivariate normality with superior performance in high dimensions.

Contribution

The paper proposes a new entropy-based estimator for KL divergence using kNN methods and develops a Gaussian goodness-of-fit test with proven consistency and improved power.

Findings

Estimator shows consistent convergence properties.

Test accurately controls Type I error.

Demonstrates superior power in high-dimensional settings.

Abstract

We examine the estimation of the Kullback-Leibler (KL) divergence and the use of the goodness-of-fit test for multivariate continuous distributions. Our starting point is the maximum entropy principle for Shannon entropy: among all distributions with a fixed mean vector and covariance matrix, the multivariate Gaussian distributions uniquely maximize entropy. As a result, the KL divergence from a moment-matched Gaussian distribution to an unknown density can then be written as the \emph{entropy difference}, which is a suitable information-theoretic measure of divergence from the Gaussian distribution. To estimate, we use $k$-nearest neighbor (kNN) estimators based on Shannon entropy and KL divergence derived from the Kozachenko-Leonenko approach and subsequent improvements, along with the consistency and $L^{2}$-convergence results established for these estimators. Motivated by previous entropy-based goodness-of-fit ideas developed for R\'enyi-type functionals under generalized Gaussian and Student-type models, we describe a KL-based test statistic as being the difference between (i) the entropy of a Gaussian model fitted to the sample mean and covariance and (ii) the KL divergence between the unknown entropy and the kNN estimate. The statistic converges to zero under multivariate normality and converges to a strictly positive bound under non-Gaussian alternatives. Results from Monte Carlo simulations on various dimensions and sample sizes indicate that the proposed procedure achieves accurate Type I error control and accurate, generally superior power compared to conventional multivariate tests of normality, particularly at medium and high dimensions.