Randomization Tests to Assess Covariate Balance When Designing and Analyzing Matched Datasets

TL;DR

This paper introduces a randomization test to evaluate if matched observational data mimics specific experimental designs, aiding in choosing appropriate analysis methods and improving causal inference accuracy.

Contribution

It develops a flexible randomization test for assessing the experimental design approximation of matched datasets, including graphical comparisons and analysis recommendations.

Findings

Test effectively detects violations of randomization assumptions.

Matched datasets often resemble rerandomization designs.

Design assumptions influence the validity of causal inferences.

Abstract

Causal analyses for observational studies are often complicated by covariate imbalances among treatment groups, and matching methodologies alleviate this complication by finding subsets of treatment groups that exhibit covariate balance. It is widely agreed upon that covariate balance can serve as evidence that a matched dataset approximates a randomized experiment, but what kind of experiment does a matched dataset approximate? In this work, we develop a randomization test for the hypothesis that a matched dataset approximates a particular experimental design, such as complete randomization, block randomization, or rerandomization. Our test can incorporate any experimental design, and it allows for a graphical display that puts several designs on the same univariate scale, thereby allowing researchers to pinpoint which design -- if any -- is most appropriate for a matched dataset. After researchers determine a plausible design, we recommend a randomization-based approach for analyzing the matched data, which can incorporate any design and treatment effect estimator. Through simulation, we find that our test can frequently detect violations of randomized assignment that harm inferential results. Furthermore, through simulation and a real application in political science, we find that matched datasets with high levels of covariate balance tend to approximate balance-constrained designs like rerandomization, and analyzing them as such can lead to precise causal analyses. However, assuming a precise design should be proceeded with caution, because it can harm inferential results if there are still substantial biases due to remaining imbalances after matching. Our approach is implemented in the randChecks R package, available on CRAN.