Design-based H\'ajek estimation for clustered and stratified experiments

TL;DR

This paper develops a new design-based H"ajek estimator and variance estimator for clustered and stratified experiments, improving causal inference accuracy in small-sample and complex designs.

Contribution

It introduces a consistent variance estimator for H"ajek estimation in clustered and stratified settings, extending the estimator to include covariates and adapt to small-$n$ designs.

Findings

The new variance estimator is consistent and recommended for mixed small and large strata.

A hybrid of the new and Neyman's estimators improves coverage in small-$n$ experiments.

Simulations confirm the method's effectiveness in heterogeneous treatment effect scenarios.

Abstract

Random allocation is essential for causal inference, but practical constraints often require assigning participants in clusters. They may be stratified pre-assignment, either of necessity or to reduce differences between treatment and control groups; but combining clustered assignment with blocking into pairs, triples, or other fine strata makes otherwise equivalent estimators perform quite differently. The two-way ANOVA with block effects can be inconsistent, as can another popular, seemingly innocuous estimator. In contrast, H\'ajek estimation remains broadly consistent for sample average treatment effects, but lacks a design-based standard error applicable with clusters and fine strata. To fill this gap, we offer a new variance estimator and establish its consistency. Analytic and simulation results recommend a hybrid of it and Neyman's estimator for designs with both small and large strata. We extend the H\'ajek estimator to accommodate covariates and adapt variance estimators to inherit Neyman-style conservativeness, at least for hypothesis testing. Further simulations suggest that with heterogeneous treatment effects, our combination of novelties is necessary and sufficient to maintain coverage in small-$n$ designs; the relevant $n$ being that of clusters, many large-scale studies are small-$n$. We consider two: a paired, aggregate-data nutritional study and an education study with student covariates and varying block sizes.