Statistical tests for whether a given set of independent, identically distributed draws does not come from a specified probability density

TL;DR

This paper evaluates statistical tests to determine if a set of i.i.d. samples originates from a specific probability density, highlighting limitations of classical methods and proposing alternative tests that better detect discrepancies in low-density regions.

Contribution

The paper introduces new statistical tests that address the smoothing issues of classical methods like Kolmogorov-Smirnov, improving detection of discrepancies in regions with small probability density.

Findings

Classical tests often miss discrepancies in low-density regions.

Proposed tests are more sensitive to small probability regions.

New methods complement existing goodness-of-fit tests.

Abstract

We discuss several tests for whether a given set of independent and identically distributed (i.i.d.) draws does not come from a specified probability density function. The most commonly used are Kolmogorov-Smirnov tests, particularly Kuiper's variant, which focus on discrepancies between the cumulative distribution function for the specified probability density and the empirical cumulative distribution function for the given set of i.i.d. draws. Unfortunately, variations in the probability density function often get smoothed over in the cumulative distribution function, making it difficult to detect discrepancies in regions where the probability density is small in comparison with its values in surrounding regions. We discuss tests without this deficiency, complementing the classical methods. The tests of the present paper are based on the plain fact that it is unlikely to draw a random number whose probability is small, provided that the draw is taken from the same distribution used in calculating the probability (thus, if we draw a random number whose probability is small, then we can be confident that we did not draw the number from the same distribution used in calculating the probability).