Limitations of Randomization Tests in Finite Samples

TL;DR

This paper explores the fundamental limitations of randomization tests in finite samples, showing many common null hypotheses do not admit exact tests under typical conditions.

Contribution

It provides a necessary and sufficient condition for null hypotheses to admit randomization tests and characterizes which nulls are feasible in finite samples.

Findings

Many common nulls, including mean zero, do not admit randomization tests.

Admissible nulls with linear group actions are limited to symmetric or Gaussian distributions.

Impossibility results are derived for continuous supports, confirming inherent limitations.

Abstract

Randomization tests deliver exact finite-sample Type 1 error control when the null satisfies the randomization hypothesis. In practice, achieving these guarantees often requires stronger conditions than the null hypothesis of primary interest. For example, sign-change tests of mean zero require symmetry and need not control finite-sample size for non-symmetric mean-zero distributions. We investigate whether the mismatch between the null and the invariance conditions required for exactness reflects the use of particular transformations or a more fundamental limitation. We provide a simple necessary and sufficient condition for a null hypothesis to admit a randomization test. Applying this framework to one-sample problems, we characterize the nulls that admit randomization tests on finite supports and derive impossibility results on continuous supports. In particular, we show that several common nulls, including mean zero, do not admit randomization tests. We further show that, among one-sample tests using linear group actions, the admissible nulls are limited to subsets of symmetric or Gaussian distributions. These results confirm that the absence of exact finite-sample validity is inherent for many commonly studied nulls and that practitioners using existing tests are not foregoing feasible exact alternatives.