What is a randomization test?

TL;DR

This paper clarifies the concept of randomization tests in statistics, introduces the term "quasi-randomization test" to distinguish theoretical models from physical randomization, and provides a unified framework for understanding various related tests.

Contribution

It introduces the concept of quasi-randomization tests, develops a general framework for conditional randomization tests, and clarifies the distinction between different types of significance tests in statistics.

Findings

Distinction between randomization and quasi-randomization tests clarified.

Framework for conditional randomization tests developed.

Application to various tests like Fisher's exact test and permutation tests demonstrated.

Abstract

The meaning of randomization tests has become obscure in statistics education and practice over the last century. This article makes a fresh attempt at rectifying this core concept of statistics. A new term -- "quasi-randomization test" -- is introduced to define significance tests based on theoretical models and distinguish these tests from the "randomization tests" based on the physical act of randomization. The practical importance of this distinction is illustrated through a real stepped-wedge cluster-randomized trial. Building on the recent literature of randomization inference, a general framework of conditional randomization tests is developed and some practical methods to construct conditioning events are given. The proposed terminology and framework are then applied to understand several widely used (quasi-)randomization tests, including Fisher's exact test, permutation tests for treatment effect, quasi-randomization tests for independence and conditional independence, adaptive randomization, and conformal prediction.