On Determining the Distribution of a Goodness-of-Fit Test Statistic

TL;DR

This paper discusses the challenges of determining the distribution of goodness-of-fit test statistics for models with unknown parameters, proposing Bayesian methods to improve accuracy over traditional bootstrap approaches.

Contribution

It introduces a Bayesian framework for more reliable goodness-of-fit testing, addressing limitations of existing bootstrap methods especially in small samples.

Findings

Bayesian methods can improve test accuracy in small samples.

Parametric bootstrap may have shortcomings in certain scenarios.

Objective Bayes techniques help achieve correct test size.

Abstract

We consider the problem of goodness-of-fit testing for a model that has at least one unknown parameter that cannot be eliminated by transformation. Examples of such problems can be as simple as testing whether a sample consists of independent Gamma observations, or whether a sample consists of independent Generalised Pareto observations given a threshold. Over time the approach to determining the distribution of a test statistic for such a problem has moved towards on-the-fly calculation post observing a sample. Modern approaches include the parametric bootstrap and posterior predictive checks. We argue that these approaches are merely approximations to integrating over the posterior predictive distribution that flows naturally from a given model. Further, we attempt to demonstrate that shortcomings which may be present in the parametric bootstrap, especially in small samples, can be reduced through the use of objective Bayes techniques, in order to more reliably produce a test with the correct size.