Another look at the Lady Tasting Tea and differences between permutation tests and randomization tests
Jesse Hemerik, Jelle J. Goeman

TL;DR
This paper clarifies the conceptual differences between permutation and randomization tests, emphasizing the importance of group structure in permutation tests and highlighting that Fisher's Lady Tasting Tea experiment is a randomization test, not a permutation test.
Contribution
It provides a clear mathematical distinction between permutation and randomization tests and discusses implications for experimental design and statistical inference.
Findings
Permutation tests require a group structure in the set of permutations.
Randomization tests do not require a group structure and can use more flexible schemes.
Using larger sets of treatment patterns improves p-value resolution and test power.
Abstract
The statistical literature is known to be inconsistent in the use of the terms "permutation test" and "randomization test". Several authors succesfully argue that these terms should be used to refer to two distinct classes of tests and that there are major conceptual differences between these classes. The present paper explains an important difference in mathematical reasoning between these classes: a permutation test fundamentally requires that the set of permutations has a group structure, in the algebraic sense; the reasoning behind a randomization test is not based on such a group structure and it is possible to use an experimental design that does not correspond to a group. In particular, we can use a randomization scheme where the number of possible treatment patterns is larger than in standard experimental designs. This leads to exact \emph{p}-values of improved resolution,…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
