The LOOP Estimator: Adjusting for Covariates in Randomized Experiments

TL;DR

The paper introduces the LOOP estimator, a new method for adjusting covariates in randomized experiments that is unbiased, leverages machine learning for variable selection, and improves upon traditional adjustment techniques.

Contribution

It proposes the LOOP estimator, which uses leave-one-out predictions with machine learning to adjust for covariates without bias, enhancing analysis of randomized trials.

Findings

LOOP estimator is unbiased under the Neyman-Rubin model.

It performs at least as well as unadjusted estimators.

Enables automatic variable selection with random forests.

Abstract

When conducting a randomized controlled trial, it is common to specify in advance the statistical analyses that will be used to analyze the data. Typically these analyses will involve adjusting for small imbalances in baseline covariates. However, this poses a dilemma, since adjusting for too many covariates can hurt precision more than it helps, and it is often unclear which covariates are predictive of outcome prior to conducting the experiment. For example, both post-stratification and OLS regression adjustments can actually increase variance (relative to a simple difference in means) if too many covariates are used. OLS is also biased under the Neyman-Rubin model. In this paper, we introduce the LOOP ("Leave-One-Out Potential outcomes") estimator of the average treatment effect. We leave out each observation and then impute that observation's treatment and control potential outcomes using a prediction algorithm, such as a random forest. This estimator is unbiased under the Neyman-Rubin model, generally performs at least as well as the unadjusted estimator, and the experimental randomization largely justifies the statistical assumptions made. Importantly, the LOOP estimator also enables us to take advantage of automatic variable selection when using random forests.