Randomization Tests for Weak Null Hypotheses in Randomized Experiments

TL;DR

This paper develops a model-free Fisher randomization test for weak null hypotheses in randomized experiments, allowing valid inference on average treatment effects without assuming treatment effect homogeneity.

Contribution

It introduces a novel imputation method for missing potential outcomes and advocates a studentized statistic, enabling exact finite-sample tests under sharp nulls and conservative large-sample tests under weak nulls.

Findings

The proposed FRT is model-free and exact under the sharp null.

It conservatively controls Type I error under the weak null.

The method extends to factorial, stratified, and clustered experiments.

Abstract

The Fisher randomization test (FRT) is appropriate for any test statistic, under a sharp null hypothesis that can recover all missing potential outcomes. However, it is often sought after to test a weak null hypothesis that the treatment does not affect the units on average. To use the FRT for a weak null hypothesis, we must address two issues. First, we need to impute the missing potential outcomes although the weak null hypothesis cannot determine all of them. Second, we need to choose a proper test statistic. For a general weak null hypothesis, we propose an approach to imputing missing potential outcomes under a compatible sharp null hypothesis. Building on this imputation scheme, we advocate a studentized statistic. The resulting FRT has multiple desirable features. First, it is model-free. Second, it is finite-sample exact under the sharp null hypothesis that we use to impute the potential outcomes. Third, it conservatively controls large-sample type I errors under the weak null hypothesis of interest. Therefore, our FRT is agnostic to the treatment effect heterogeneity. We establish a unified theory for general factorial experiments. We also extend it to stratified and clustered experiments.