Optimal Stratification of Survey Experiments

TL;DR

This paper introduces a new family of stratified experimental designs that improve efficiency in treatment effect estimation by balancing sampling and assignment, applicable even with heterogeneous costs and limited budgets.

Contribution

It develops a generalized stratification method for two-stage experiments, optimizing design under cost constraints and providing new inference techniques for enhanced efficiency.

Findings

Variance of treatment effect estimation is reduced by the proposed stratification.

Optimal design heuristics improve efficiency in budget-constrained experiments.

Application to economics papers demonstrates practical benefits.

Abstract

This paper studies a two-stage model of experimentation, where the researcher first samples representative units from an eligible pool, then assigns each sampled unit to treatment or control. To implement balanced sampling and assignment, we introduce a new family of finely stratified designs that generalize matched pairs randomization to propensities p(x) not equal to 1/2. We show that two-stage stratification nonparametrically dampens the variance of treatment effect estimation. We formulate and solve the optimal stratification problem with heterogeneous costs and fixed budget, providing simple heuristics for the optimal design. In settings with pilot data, we show that implementing a consistent estimate of this design is also efficient, minimizing asymptotic variance subject to the budget constraint. We also provide new asymptotically exact inference methods, allowing experimenters to fully exploit the efficiency gains from both stratified sampling and assignment. An application to nine papers recently published in top economics journals demonstrates the value of our methods.