Design-Based Variance Estimation for Modern Heterogeneity-Robust Difference-in-Differences Estimators

TL;DR

This paper develops a design-based variance estimation method for modern heterogeneity-robust difference-in-differences estimators, demonstrating its effectiveness through simulations and an NHANES case study.

Contribution

It introduces a variance estimation approach that accounts for complex survey designs, improving inference accuracy for modern DiD estimators.

Findings

Standard stratified-cluster variance formula is consistent under regularity conditions.

Ignoring survey design leads to severely underestimated standard errors.

Using PSU-level clustering with survey weights achieves near-nominal coverage.

Abstract

Modern heterogeneity-robust difference-in-differences estimators derive their asymptotic properties under iid, cluster, or fixed-design frameworks that abstract from complex survey sampling, yet practitioners routinely apply them to nationally representative surveys with stratified cluster designs. We show that, under standard regularity conditions, the influence functions of each smooth IF-based or regression-based modern DiD estimator satisfy Binder's (1983) smoothness conditions, so the standard stratified-cluster variance formula applied to their values produces design-consistent standard errors. A Monte Carlo study with 66,000 replications shows where the design effect comes from. HC1 standard errors that treat observations as iid produce coverage as low as 34% under a baseline survey design and below 11% under informative sampling. Combining the survey-weighted point estimate with PSU-level clustering - the practitioner's cluster=psu heuristic - recovers near-nominal coverage across all scenarios. Adding strata and finite-population corrections yields incremental precision but is not required for valid coverage. Survey-weighted doubly robust estimation produces well-calibrated inference when parallel trends hold only conditionally. An NHANES illustration of the ACA dependent coverage provision shows that point estimates and standard errors change substantively - enough to reverse significance conclusions - when the survey design is accounted for. We provide diff-diff (https://github.com/igerber/diff-diff), an open-source Python package implementing design-based variance for fifteen modern DiD estimators.