A conditional randomization test to account for covariate imbalance in randomized experiments

TL;DR

This paper introduces a conditional randomization test that adjusts for covariate imbalance in randomized experiments, ensuring correct significance levels and improving validity when covariates are collected post-randomization.

Contribution

It formalizes covariate balance using new notation, proves the test's correctness, and demonstrates its effectiveness through simulations and a real marketing experiment.

Findings

Conditional randomization test maintains correct significance level.

Test behaves like traditional covariate adjustment methods.

Effective in scenarios with post-randomization covariate collection.

Abstract

We consider the conditional randomization test as a way to account for covariate imbalance in randomized experiments. The test accounts for covariate imbalance by comparing the observed test statistic to the null distribution of the test statistic conditional on the observed covariate imbalance. We prove that the conditional randomization test has the correct significance level and introduce original notation to describe covariate balance more formally. Through simulation, we verify that conditional randomization tests behave like more traditional forms of covariate adjustmet but have the added benefit of having the correct conditional significance level. Finally, we apply the approach to a randomized product marketing experiment where covariate information was collected after randomization.