Inference in experiments conditional on observed imbalances in covariates

TL;DR

This paper analyzes the properties of difference-in-means and OLS estimators when conditioning on observed covariate imbalances in randomized trials, providing guidance for inference in such scenarios.

Contribution

It offers a theoretical analysis of conditional inference methods, including a new Fisher's exact test variant, for experiments with covariate imbalances.

Findings

Conditional estimators have specific statistical properties.

Guidance on handling covariate imbalances in inference.

Introduction of a new Fisher's exact test variant.

Abstract

Double blind randomized controlled trials are traditionally seen as the gold standard for causal inferences as the difference-in-means estimator is an unbiased estimator of the average treatment effect in the experiment. The fact that this estimator is unbiased over all possible randomizations does not, however, mean that any given estimate is close to the true treatment effect. Similarly, while pre-determined covariates will be balanced between treatment and control groups on average, large imbalances may be observed in a given experiment and the researcher may therefore want to condition on such covariates using linear regression. This paper studies the theoretical properties of both the difference-in-means and OLS estimators \emph{conditional} on observed differences in covariates. By deriving the statistical properties of the conditional estimators, we can establish guidance for how to deal with covariate imbalances. We study both inference with OLS, as well as with a new version of Fisher's exact test, where the randomization distribution comes from a small subset of all possible assignment vectors.