Scalable Hash-Based Estimation of Divergence Measures

TL;DR

This paper introduces a scalable hashing-based method for estimating divergence measures between continuous distributions, achieving optimal accuracy and computational efficiency, especially for high-dimensional data.

Contribution

It presents the first empirical divergence estimator that is both computationally optimal and achieves the parametric MSE rate, based on locality sensitive hashing.

Findings

Estimator converges to true f-divergences with high accuracy.

Achieves optimal MSE rate of O(1/N) for certain smoothness classes.

Computational complexity is linear, O(N), making it scalable for large datasets.

Abstract

We propose a scalable divergence estimation method based on hashing. Consider two continuous random variables $X$ and $Y$ whose densities have bounded support. We consider a particular locality sensitive random hashing, and consider the ratio of samples in each hash bin having non-zero numbers of Y samples. We prove that the weighted average of these ratios over all of the hash bins converges to f-divergences between the two samples sets. We show that the proposed estimator is optimal in terms of both MSE rate and computational complexity. We derive the MSE rates for two families of smooth functions; the H\"{o}lder smoothness class and differentiable functions. In particular, it is proved that if the density functions have bounded derivatives up to the order $d/2$, where $d$ is the dimension of samples, the optimal parametric MSE rate of $O(1/N)$ can be achieved. The computational complexity is shown to be $O(N)$, which is optimal. To the best of our knowledge, this is the first empirical divergence estimator that has optimal computational complexity and achieves the optimal parametric MSE estimation rate.