Multivariate f-Divergence Estimation With Confidence

TL;DR

This paper proves the asymptotic normality of a new ensemble estimator for f-divergence, enabling reliable inference in high-dimensional settings and improving understanding of divergence estimation's statistical properties.

Contribution

It establishes the asymptotic normality of a recent ensemble f-divergence estimator, providing theoretical guarantees and practical tools for divergence-based inference.

Findings

Estimator has MSE rate of O(1/T)

Estimator performs well in high dimensions

Theoretical results validated experimentally

Abstract

The problem of f-divergence estimation is important in the fields of machine learning, information theory, and statistics. While several nonparametric divergence estimators exist, relatively few have known convergence properties. In particular, even for those estimators whose MSE convergence rates are known, the asymptotic distributions are unknown. We establish the asymptotic normality of a recently proposed ensemble estimator of f-divergence between two distributions from a finite number of samples. This estimator has MSE convergence rate of O(1/T), is simple to implement, and performs well in high dimensions. This theory enables us to perform divergence-based inference tasks such as testing equality of pairs of distributions based on empirical samples. We experimentally validate our theoretical results and, as an illustration, use them to empirically bound the best achievable classification error.