Generalized Exponential Concentration Inequality for R\'enyi Divergence Estimation

TL;DR

This paper derives a finite sample exponential inequality bound for Re9nyi divergence estimators, advancing the theoretical understanding of divergence estimation in nonparametric statistics with practical numerical validation.

Contribution

It provides the first finite sample exponential inequality bound for Re9nyi divergence estimators in a nonparametric setting, specifically for smooth densities.

Findings

Established a finite sample exponential inequality bound for Re9nyi divergence estimation.

Validated theoretical results through numerical experiments.

Enhanced understanding of divergence estimator convergence in high-dimensional settings.

Abstract

Estimating divergences in a consistent way is of great importance in many machine learning tasks. Although this is a fundamental problem in nonparametric statistics, to the best of our knowledge there has been no finite sample exponential inequality convergence bound derived for any divergence estimators. The main contribution of our work is to provide such a bound for an estimator of R\'enyi-$\alpha$ divergence for a smooth H\"older class of densities on the $d$-dimensional unit cube $[0, 1]^d$. We also illustrate our theoretical results with a numerical experiment.