Rerandomization for covariate balance mitigates p-hacking in regression adjustment

TL;DR

This paper demonstrates that rerandomization in experimental design reduces false positives caused by p-hacking, especially when using strict thresholds, thereby improving the reliability of treatment effect estimates.

Contribution

It provides a theoretical framework showing rerandomization mitigates p-hacking effects and guides threshold selection for practical implementation.

Findings

Rerandomization reduces false discoveries from p-hacking.

Stringent rerandomization thresholds effectively resolve p-hacking.

Guidance on choosing rerandomization thresholds in practice.

Abstract

Rerandomization enforces covariate balance across treatment groups in the design stage of experiments. Despite its intuitive appeal, its theoretical justification remains unsatisfying because its benefits of improving efficiency for estimating the average treatment effect diminish if we use regression adjustment in the analysis stage. To strengthen the theory of rerandomization, we show that it mitigates false discoveries resulting from $p$-hacking, the practice of strategically selecting covariates to get more significant $p$-values. Moreover, we show that rerandomization with a sufficiently stringent threshold can resolve $p$-hacking. As a byproduct, our theory offers guidance for choosing the threshold in rerandomization in practice.