The Optimality of Blocking Designs in Equally and Unequally Allocated Randomized Experiments with General Response

TL;DR

This paper investigates the optimality of blocking designs in randomized experiments with various response types, showing Fisher's blocking design is asymptotically optimal in balancing randomness and efficiency.

Contribution

It establishes the asymptotic optimality of Fisher's blocking design under general response models and compares it to other allocation strategies, including complete randomization and perfect balance.

Findings

Complete randomization is minimax in the Neyman model.

Deterministic perfect-balance allocation is optimal in the population model but NP-hard to compute.

Fisher's blocking design achieves asymptotic optimality in the tail criterion.

Abstract

We consider the performance of the difference-in-means estimator in a two-arm randomized experiment under common experimental endpoints such as continuous (regression), incidence, proportion and survival. We examine performance under both equal and unequal allocation to treatment groups and we consider both the Neyman randomization model and the population model. We show that in the Neyman model, where the only source of randomness is the treatment manipulation, there is no free lunch: complete randomization is minimax for the estimator's mean squared error. In the population model, where each subject experiences response noise with zero mean, the optimal design is the deterministic perfect-balance allocation. However, this allocation is generally NP-hard to compute and moreover, depends on unknown response parameters. When considering the tail criterion of Kapelner et al. (2021), we show the optimal design is less random than complete randomization and more random than the deterministic perfect-balance allocation. We prove that Fisher's blocking design provides the asymptotically optimal degree of experimental randomness. Theoretical results are supported by simulations in all considered experimental settings.