Robust Kullback-Leibler Divergence and Universal Hypothesis Testing for Continuous Distributions

TL;DR

This paper introduces a robust Kullback-Leibler divergence based on Levy balls to address the discontinuity issues in continuous distributions, enabling asymptotically optimal universal hypothesis testing.

Contribution

It proposes a new robust KLD measure that is continuous under weak convergence, facilitating universal hypothesis testing for continuous distributions.

Findings

Robust KLD is continuous in the distribution function.

Developed a universal hypothesis test for continuous data.

Test is asymptotically optimal, similar to Hoeffding's test for discrete data.

Abstract

Universal hypothesis testing refers to the problem of deciding whether samples come from a nominal distribution or an unknown distribution that is different from the nominal distribution. Hoeffding's test, whose test statistic is equivalent to the empirical Kullback-Leibler divergence (KLD), is known to be asymptotically optimal for distributions defined on finite alphabets. With continuous observations, however, the discontinuity of the KLD in the distribution functions results in significant complications for universal hypothesis testing. This paper introduces a robust version of the classical KLD, defined as the KLD from a distribution to the L'evy ball of a known distribution. This robust KLD is shown to be continuous in the underlying distribution function with respect to the weak convergence. The continuity property enables the development of a universal hypothesis test for continuous observations that is shown to be asymptotically optimal for continuous distributions in the same sense as that of the Hoeffding's test for discrete distributions.