A randomization-based theory for preliminary testing of covariate balance in controlled trials

TL;DR

This paper analyzes the validity and efficiency of preliminary covariate balance tests in randomized trials, showing that such tests can lead to less efficient or overly conservative inferences, and recommends against their use.

Contribution

It provides a theoretical and simulation-based evaluation of covariate adjustment strategies after preliminary balance testing, highlighting their limitations and advising against their use.

Findings

Preliminary-test covariate adjustment can be less efficient than unadjusted methods.

Such adjustment may produce anticonservative confidence intervals.

Overconservative intervals result from fully interacted specification adjustments.

Abstract

Randomized trials balance all covariates on average and provide the gold standard for estimating treatment effects. Chance imbalances nevertheless exist more or less in realized treatment allocations and intrigue an important question: what should we do in case the treatment groups differ with respect to some important baseline characteristics? A common strategy is to conduct a {\it preliminary test} of the balance of baseline covariates after randomization, and invoke covariate adjustment for subsequent inference if and only if the realized allocation fails some prespecified criterion. Although such practice is intuitive and popular among practitioners, the existing literature has so far only evaluated its properties under strong parametric model assumptions in theory and simulation, yielding results of limited generality. To fill this gap, we examine two strategies for conducting preliminary test-based covariate adjustment by regression, and evaluate the validity and efficiency of the resulting inferences from the randomization-based perspective. As it turns out, the preliminary-test estimator based on the analysis of covariance can be even less efficient than the unadjusted difference in means, and risks anticonservative confidence intervals based on normal approximation even with the robust standard error. The preliminary-test estimator based on the fully interacted specification is on the other hand less efficient than its counterpart under the {\it always-adjust} strategy, and yields overconservative confidence intervals based on normal approximation. Based on theory and simulation, we echo the existing literature and do not recommend the preliminary-test procedure for covariate adjustment in randomized trials.