Properties of restricted randomization with implications for experimental design

TL;DR

This paper examines the risks of heavily restricted randomization in experimental design, introduces a combinatoric diagnostic measure, and discusses how to balance covariate balance with estimator accuracy.

Contribution

It formalizes the risk of high mean squared error in restricted randomization and proposes a new combinatoric-based diagnostic tool to evaluate and mitigate this risk.

Findings

Variance of MSE is linearly related to the diagnostic measure.

Restricted designs can increase the risk of high MSE.

The diagnostic measure helps identify potentially problematic designs.

Abstract

Recently, there as been an increasing interest in the use of heavily restricted randomization designs which enforces balance on observed covariates in randomized controlled trials. However, when restrictions are strict, there is a risk that the treatment effect estimator will have a very high mean squared error. In this paper, we formalize this risk and propose a novel combinatoric-based approach to describe and address this issue. First, we validate our new approach by re-proving some known properties of complete randomization and restricted randomization. Second, we propose a novel diagnostic measure for restricted designs that only use the information embedded in the combinatorics of the design. Third, we show that the variance of the mean squared error of the difference-in-means estimator in a randomized experiment is a linear function of this diagnostic measure. Finally, we identify situations in which restricted designs can lead to an increased risk of getting a high mean squared error and discuss how our diagnostic measure can be used to detect such designs. Our results have implications for any restricted randomization design and can be used to evaluate the trade-off between enforcing balance on observed covariates and avoiding too restrictive designs.