Approximating $f$-Divergences with Rank Statistics

TL;DR

This paper introduces a novel rank-statistic method for approximating $f$-divergences that avoids explicit density estimation, providing theoretical guarantees and practical validation in high-dimensional settings.

Contribution

It proposes a rank-based divergence estimator with proven convergence properties, extending to high dimensions via slicing, and demonstrates its effectiveness through empirical benchmarks.

Findings

Estimator is monotone in $K$ and always a lower bound of the true divergence.

Provides convergence rates and finite-sample deviation bounds.

Validated through neural benchmarks and generative modeling experiments.

Abstract

We introduce a rank-statistic approximation of $f$-divergences that avoids explicit density-ratio estimation by working directly with the distribution of ranks. For a resolution parameter $K$, we map the mismatch between two univariate distributions $\mu$ and $\nu$ to a rank histogram on $\{ 0, \ldots, K\}$ and measure its deviation from uniformity via a discrete $f$-divergence, yielding a rank-statistic divergence estimator. We prove that the resulting estimator of the divergence is monotone in $K$, is always a lower bound of the true $f$-divergence, and we establish quantitative convergence rates for $K\to\infty$ under mild regularity of the quantile-domain density ratio. To handle high-dimensional data, we define the sliced rank-statistic $f$-divergence by averaging the univariate construction over random projections, and we provide convergence results for the sliced limit as well. We also derive finite-sample deviation bounds along with asymptotic normality results for the estimator. Finally, we empirically validate the approach by benchmarking against neural baselines and illustrating its use as a learning objective in generative modelling experiments.