A Concentration Result of Estimating Phi-Divergence using Data Dependent Partition

TL;DR

This paper introduces a new multi-variate data-dependent partitioning method for estimating phi-divergence between unknown distributions, providing convergence rates under power law decay assumptions.

Contribution

It generalizes existing methods to multivariate cases and establishes convergence rates for divergence estimation with sample complexity bounds.

Findings

Convergence rate established for the proposed method.

Sample complexity bounds depend on the power law decay.

Method effectively estimates divergence with bounded error.

Abstract

Estimation of the $\phi$-divergence between two unknown probability distributions using empirical data is a fundamental problem in information theory and statistical learning. We consider a multi-variate generalization of the data dependent partitioning method for estimating divergence between the two unknown distributions. Under the assumption that the distribution satisfies a power law of decay, we provide a convergence rate result for this method on the number of samples and hyper-rectangles required to ensure the estimation error is bounded by a given level with a given probability.