Deviation Inequalities for R\'{e}nyi Divergence Estimators via Variational Expression

TL;DR

This paper develops exponential deviation inequalities for Rényi divergence estimators, providing probabilistic bounds without restrictive assumptions, with applications in information theory, privacy auditing, and hypothesis testing.

Contribution

It introduces new deviation bounds for Rényi divergence estimators that do not require compact support or bounded densities, advancing theoretical understanding and practical applications.

Findings

Established exponential deviation inequalities for estimators

Derived one-sided concentration bounds useful in information theory

Provided non-asymptotic guarantees for privacy testing

Abstract

R\'enyi divergences play a pivotal role in information theory, statistics, and machine learning. While several estimators of these divergences have been proposed in the literature with their consistency properties established and minimax convergence rates quantified, existing accounts of probabilistic bounds governing the estimation error are relatively underdeveloped. Here, we make progress in this regard by establishing exponential deviation inequalities for smoothed plug-in estimators and neural estimators by relating the error to an appropriate empirical process and leveraging tools from empirical process theory. In particular, our approach does not require the underlying distributions to be compactly supported or have densities bounded away from zero, an assumption prevalent in existing results. The deviation inequality also leads to a one-sided concentration bound from the expectation, which is useful in random-coding arguments over continuous alphabets in information theory with potential applications to physical-layer security. As another concrete application, we consider a hypothesis testing framework for auditing R\'{e}nyi differential privacy using the neural estimator as a test statistic and obtain non-asymptotic performance guarantees for such a test.