Inference on a New Class of Sample Average Treatment Effects

TL;DR

This paper develops new variance formulas for inference on a broad class of causal estimands in RCTs, highlighting differences from traditional SATE inference, with theoretical, simulation, and empirical validation.

Contribution

It introduces a general variance estimation framework for mixed SATT and SATC estimands, extending Robins (1988) and addressing coverage issues in SATE-based inference.

Findings

Consistent variance estimators exist for SATT and SATC.

Inference on SATE can misrepresent true coverage for SATT/SATC.

Monte Carlo simulations and empirical data validate the new methods.

Abstract

We derive new variance formulas for inference on a general class of estimands of causal average treatment effects in a Randomized Control Trial (RCT). We generalize Robins (1988) and show that when the estimand of interest is the Sample Average Treatment Effect of the Treated (SATT or SATC for controls), a consistent variance estimator exists. Although these estimands are equal to the Sample Average Treatment Effect (SATE) in expectation, potentially large differences in both accuracy and coverage can occur by the change of estimand, even asymptotically. Inference on the SATE, even using a conservative confidence interval, provides incorrect coverage of the SATT or SATC. We derive the variance and limiting distribution of a new and general class of estimands---any mixing between SATT and SATC---for which the SATE is a specific case. We demonstrate the applicability of the new theoretical results using Monte-Carlo simulations and an empirical application with hundreds of online experiments with an average sample size of approximately one hundred million observations per experiment. An R package, estCI, that implements all the proposed estimation procedures is available.