No star is good news: A unified look at rerandomization based on $p$-values from covariate balance tests

TL;DR

This paper investigates how rerandomization based on $p$-values from covariate balance tests affects the efficiency and validity of treatment effect estimators in randomized experiments, providing theoretical insights and practical recommendations.

Contribution

It analyzes the asymptotic properties of treatment effect estimators under various $p$-value based rerandomization schemes, highlighting the efficiency of fully interacted regression.

Findings

Fully interacted regression estimator is most efficient under all ReP schemes.

ReP improves covariate balance and estimator efficiency.

Standard regression analysis remains valid but may be overly conservative.

Abstract

Modern social and biomedical scientific publications require the reporting of covariate balance tables with not only covariate means by treatment group but also the associated $p$-values from significance tests of their differences. The practical need to avoid small $p$-values renders balance check and rerandomization by hypothesis testing standards an attractive tool for improving covariate balance in randomized experiments. Despite the intuitiveness of such practice and its arguably already widespread use in reality, the existing literature knows little about its implications on subsequent inference, subjecting many effectively rerandomized experiments to possibly inefficient analyses. To fill this gap, we examine a variety of potentially useful schemes for rerandomization based on $p$-values (ReP) from covariate balance tests, and demonstrate their impact on subsequent inference. Specifically, we focus on three estimators of the average treatment effect from the unadjusted, additive, and fully interacted linear regressions of the outcome on treatment, respectively, and derive their respective asymptotic sampling properties under ReP. The main findings are twofold. First, the estimator from the fully interacted regression is asymptotically the most efficient under all ReP schemes examined, and permits convenient regression-assisted inference identical to that under complete randomization. Second, ReP improves not only covariate balance but also the efficiency of the estimators from the unadjusted and additive regressions asymptotically. The standard regression analysis, in consequence, is still valid but can be overly conservative.