Ensemble estimation of multivariate f-divergence

TL;DR

This paper introduces a new ensemble estimator for multivariate f-divergence that achieves a fast convergence rate of O(1/T), improving performance in high-dimensional settings and validated through experiments.

Contribution

It derives the first known convergence rate for a density plug-in f-divergence estimator and proposes a simple, high-performing ensemble estimator with optimal convergence.

Findings

The ensemble estimator attains an O(1/T) convergence rate.

Experimental results confirm the theoretical advantages.

The method performs well in high-dimensional scenarios.

Abstract

f-divergence estimation is an important problem in the fields of information theory, machine learning, and statistics. While several divergence estimators exist, relatively few of their convergence rates are known. We derive the MSE convergence rate for a density plug-in estimator of f-divergence. Then by applying the theory of optimally weighted ensemble estimation, we derive a divergence estimator with a convergence rate of O(1/T) that is simple to implement and performs well in high dimensions. We validate our theoretical results with experiments.