Graphical Test for Discrete Uniformity and its Applications in Goodness of Fit Evaluation and Multiple Sample Comparison

TL;DR

This paper introduces new graphical and statistical methods using confidence bands for the empirical CDF of PIT values to assess uniformity and compare multiple samples, aiding goodness-of-fit and convergence diagnostics.

Contribution

It presents novel simulation and optimization techniques for constructing simultaneous confidence bands for ECDFs, applicable to goodness-of-fit testing and multiple sample comparison.

Findings

Effective graphical test for uniformity using confidence bands.

Applicable to finite reference samples in goodness-of-fit.

Useful in MCMC convergence diagnostics.

Abstract

Assessing goodness of fit to a given distribution plays an important role in computational statistics. The Probability integral transformation (PIT) can be used to convert the question of whether a given sample originates from a reference distribution into a problem of testing for uniformity. We present new simulation and optimization based methods to obtain simultaneous confidence bands for the whole empirical cumulative distribution function (ECDF) of the PIT values under the assumption of uniformity. Simultaneous confidence bands correspond to such confidence intervals at each point that jointly satisfy a desired coverage. These methods can also be applied in cases where the reference distribution is represented only by a finite sample. The confidence bands provide an intuitive ECDF-based graphical test for uniformity, which also provides useful information on the quality of the discrepancy. We further extend the simulation and optimization methods to determine simultaneous confidence bands for testing whether multiple samples come from the same underlying distribution. This multiple sample comparison test is especially useful in Markov chain Monte Carlo convergence diagnostics. We provide numerical experiments to assess the properties of the tests using both simulated and real world data and give recommendations on their practical application in computational statistics workflows.