Conditional As-If Analyses in Randomized Experiments

TL;DR

This paper investigates when it is valid to analyze randomized experiments as if they were conducted under different designs, providing theoretical justification and new methodological insights within a randomization-based framework.

Contribution

It establishes conditions under which analyzing an experiment as if it had a different randomization design is legitimate, using conditioning principles.

Findings

A sufficient condition for valid analysis is that the alternative design is derived by conditioning on the original.

The theory justifies some existing analysis methods and questions others.

It suggests new approaches like conditioning on covariate balance.

Abstract

The injunction to `analyze the way you randomize' is well-known to statisticians since Fisher advocated for randomization as the basis of inference. Yet even those convinced by the merits of randomization-based inference seldom follow this injunction to the letter. Bernoulli randomized experiments are often analyzed as completely randomized experiments, and completely randomized experiments are analyzed as if they had been stratified; more generally, it is not uncommon to analyze an experiment as if it had been randomized differently. This paper examines the theoretical foundation behind this practice within a randomization-based framework. Specifically, we ask when is it legitimate to analyze an experiment randomized according to one design as if it had been randomized according to some other design. We show that a sufficient condition for this type of analysis to be valid is that the design used for analysis be derived from the original design by an appropriate form of conditioning. We use our theory to justify certain existing methods, question others, and finally suggest new methodological insights such as conditioning on approximate covariate balance.