Permutation tests using arbitrary permutation distributions

TL;DR

This paper proves that permutation tests remain valid even when using arbitrary distributions over any subset of permutations, broadening their applicability beyond traditional uniform sampling and subgroup restrictions.

Contribution

It introduces a unified theoretical framework showing permutation tests are valid under arbitrary permutation distributions, not just uniform or subgroup-based ones.

Findings

Permutation p-values are valid under arbitrary permutation distributions.

Theoretical framework encompasses all known permutation test results.

Expands the practical flexibility of permutation testing methods.

Abstract

Permutation tests date back nearly a century to Fisher's randomized experiments, and remain an immensely popular statistical tool, used for testing hypotheses of independence between variables and other common inferential questions. Much of the existing literature has emphasized that, for the permutation p-value to be valid, one must first pick a subgroup $G$ of permutations (which could equal the full group) and then recalculate the test statistic on permuted data using either an exhaustive enumeration of $G$, or a sample from $G$ drawn uniformly at random. In this work, we demonstrate that the focus on subgroups and uniform sampling are both unnecessary for validity -- in fact, a simple random modification of the permutation p-value remains valid even when using an arbitrary distribution (not necessarily uniform) over any subset of permutations (not necessarily a subgroup). We provide a unified theoretical treatment of such generalized permutation tests, recovering all known results from the literature as special cases. Thus, this work expands the flexibility of the permutation test toolkit available to the practitioner.