Sampling-based randomized designs for causal inference under the potential outcomes framework

TL;DR

This paper analyzes the properties of the mean-difference estimator for causal effects in randomized experiments where sampling occurs after treatment assignment, revealing its equivalence to the traditional design and examining the limited benefits of pre-treatment measurements.

Contribution

It establishes the inferential properties of the mean-difference estimator in a reversed sampling and treatment assignment scenario, clarifying its applicability and limitations.

Findings

Inferential properties are identical to the traditional design.

Pre-treatment measurements often do not improve estimator precision.

Simulation confirms the theoretical results in a nanomaterials experiment.

Abstract

We establish the inferential properties of the mean-difference estimator for the average treatment effect in randomized experiments where each unit in a population is randomized to one of two treatments and then units within treatment groups are randomly sampled. The properties of this estimator are well-understood in the experimental design scenario where first units are randomly sampled and then treatment is randomly assigned, but not for the aforementioned scenario where the sampling and treatment assignment stages are reversed. We find that the inferential properties of the mean-difference estimator under this experimental design scenario are identical to those under the more common sample-first-randomize-second design. This finding will bring some clarifications about sampling-based randomized designs for causal inference, particularly for settings where there is a finite super-population. Finally, we explore to what extent pre-treatment measurements can be used to improve upon the mean-difference estimator for this randomize-first-sample-second design. Unfortunately, we find that pre-treatment measurements are often unhelpful in improving the precision of average treatment effect estimators under this design, unless a large number of pre-treatment measurements that are highly associative with the post-treatment measurements can be obtained. We confirm these results using a simulation study based on a real experiment in nanomaterials.