Markov Chain Monte Carlo Significance Tests

TL;DR

This paper reviews and unifies MCMC-based significance testing methods, which allow for valid p-value computation when exact sampling from the null distribution is infeasible, and discusses their applications.

Contribution

It introduces a new unifying method for MCMC significance tests, consolidating previous approaches and highlighting their practical applications.

Findings

MCMC significance tests can produce valid p-values without exact sampling.

The new method unifies existing MCMC procedures for significance testing.

Applications include goodness-of-fit, gerrymandering detection, and conditional independence tests.

Abstract

Monte Carlo significance tests are a general tool that produce p-values by generating samples from the null distribution. However, Monte Carlo tests are limited to null hypothesis which we can exactly sample from. Markov chain Monte Carlo (MCMC) significance tests are a way to produce statistical valid p-values for null hypothesis we can only approximately sample from. These methods were first introduced by Besag and Clifford in 1989 and make no assumptions on the mixing time of the MCMC procedure. Here we review the two methods of Besag and Clifford and introduce a new method that unifies the existing procedures. We use simple examples to highlight the difference between MCMC significance tests and standard Monte Carlo tests based on exact sampling. We also survey a range of contemporary applications in the literature including goodness-of-fit testing for the Rasch model, tests for detecting gerrymandering [8] and a permutation based test of conditional independence [3].