Benchmarking Estimators for Natural Experiments: A Novel Dataset and a Doubly Robust Algorithm

TL;DR

This paper introduces a new dataset and benchmark for evaluating estimators of treatment effects in natural experiments, demonstrating the superior performance of doubly robust estimators and proposing a novel estimator with improved variance properties.

Contribution

The paper provides a novel natural experiment dataset, a comprehensive benchmark for estimator evaluation, and a new doubly robust estimator with a closed-form variance expression.

Findings

Doubly robust estimators outperform other methods by orders of magnitude.

The benchmark reveals estimator performance varies with sample size, treatment correlation, and propensity score accuracy.

A new estimator with a novel loss function improves variance properties.

Abstract

Estimating the effect of treatments from natural experiments, where treatments are pre-assigned, is an important and well-studied problem. We introduce a novel natural experiment dataset obtained from an early childhood literacy nonprofit. Surprisingly, applying over 20 established estimators to the dataset produces inconsistent results in evaluating the nonprofit's efficacy. To address this, we create a benchmark to evaluate estimator accuracy using synthetic outcomes, whose design was guided by domain experts. The benchmark extensively explores performance as real world conditions like sample size, treatment correlation, and propensity score accuracy vary. Based on our benchmark, we observe that the class of doubly robust treatment effect estimators, which are based on simple and intuitive regression adjustment, generally outperform other more complicated estimators by orders of magnitude. To better support our theoretical understanding of doubly robust estimators, we derive a closed form expression for the variance of any such estimator that uses dataset splitting to obtain an unbiased estimate. This expression motivates the design of a new doubly robust estimator that uses a novel loss function when fitting functions for regression adjustment. We release the dataset and benchmark in a Python package; the package is built in a modular way to facilitate new datasets and estimators.