Insights on Variance Estimation for Blocked and Matched Pairs Designs

TL;DR

This paper examines variance estimation in blocked and matched pairs experimental designs, addressing gaps in existing methods for blocks of varying sizes, and proposes new estimators to improve accuracy across different scenarios.

Contribution

It reconciles two main literatures on variance estimation, analyzes their performance, and introduces novel estimators for blocks of different sizes, including singleton units.

Findings

Existing estimators have limitations with varying block sizes.

New variance estimators improve accuracy for mixed block sizes.

Theoretical analysis supports the proposed estimators' effectiveness.

Abstract

Evaluating blocked randomized experiments from a potential outcomes perspective has two primary branches of work. The first focuses on larger blocks, with multiple treatment and control units in each block. The second focuses on matched pairs, with a single treatment and control unit in each block. These literatures not only provide different estimators for the standard errors of the estimated average impact, but they are also built on different sets of assumptions. Neither literature handles cases with blocks of varying size that contain singleton treatment or control units, a case which can occur in a variety of contexts, such as with different forms of matching or post-stratification. In this paper, we reconcile the literatures by carefully examining the performance of variance estimators under several different frameworks. We then use these insights to derive novel variance estimators for experiments containing blocks of different sizes.