Conditioning on posterior samples for flexible frequentist goodness-of-fit testing

TL;DR

This paper introduces a novel Bayesian posterior sampling method for flexible and valid goodness-of-fit testing, overcoming limitations of existing approaches and applicable to a wider range of models.

Contribution

It proposes a new approach called aCSS-B that conditions on posterior samples, expanding the scope and effectiveness of goodness-of-fit tests.

Findings

The method is approximately valid for various models.

It outperforms existing methods on applicable models.

It enables goodness-of-fit testing where previous methods failed.

Abstract

Tests of goodness of fit are used in nearly every domain where statistics is applied. One powerful and flexible approach is to sample artificial data sets that are exchangeable with the real data under the null hypothesis (but not under the alternative), as this allows the analyst to conduct a valid test using any test statistic they desire. Such sampling is typically done by conditioning on either an exact or approximate sufficient statistic, but existing methods for doing so have significant limitations, which either preclude their use or substantially reduce their power or computational tractability for many important models. In this paper, we propose to condition on samples from a Bayesian posterior distribution, which constitute a very different type of approximate sufficient statistic than those considered in prior work. Our approach, approximately co-sufficient sampling via Bayes (aCSS-B), considerably expands the scope of this flexible type of goodness-of-fit testing. We prove the approximate validity of the resulting test, and demonstrate its utility on three common null models where no existing methods apply, as well as its outperformance on models where existing methods do apply.