Limit distribution theory for $f$-Divergences

TL;DR

This paper develops a general methodology to derive the limit distribution of $f$-divergences, enabling valid statistical inference for their estimation errors in various applications including privacy auditing.

Contribution

It introduces a unified approach using the functional delta method for deriving distributional limits of $f$-divergences, covering several prominent divergences and providing new limit theorems.

Findings

Established limit distribution results for four key $f$-divergences.

Derived one- and two-sample limit theorems under null and alternative hypotheses.

Applied the theory to differential privacy auditing for significance testing.

Abstract

$f$-divergences, which quantify discrepancy between probability distributions, are ubiquitous in information theory, machine learning, and statistics. While there are numerous methods for estimating $f$-divergences from data, a limit distribution theory, which quantifies fluctuations of the estimation error, is largely obscure. As limit theorems are pivotal for valid statistical inference, to close this gap, we develop a general methodology for deriving distributional limits for $f$-divergences based on the functional delta method and Hadamard directional differentiability. Focusing on four prominent $f$-divergences -- Kullback-Leibler divergence, $\chi^2$ divergence, squared Hellinger distance, and total variation distance -- we identify sufficient conditions on the population distributions for the existence of distributional limits and characterize the limiting variables. These results are used to derive one- and two-sample limit theorems for Gaussian-smoothed $f$-divergences, both under the null and the alternative. Finally, an application of the limit distribution theory to auditing differential privacy is proposed and analyzed for significance level and power against local alternatives.