Sequentially-Rerandomized Switchback Experiments

TL;DR

The paper introduces Sequentially-Rerandomized Switchback Experiments (SRSB), a novel design for online policy evaluation that improves efficiency and robustness over traditional methods by balancing covariates over time.

Contribution

It proposes a new experimental design, SRSB, that mitigates challenges like heterogeneity and carryover in online experiments, with theoretical and simulation validation.

Findings

SRSB enhances precision by balancing lagged outcomes and covariates.

Finite-sample and asymptotic inference methods are developed for SRSB.

Simulations show SRSB's robustness and practical advantages over standard switchback designs.

Abstract

Large-scale online platforms and marketplace systems often evaluate new policies through experiments that randomize treatment across operational units (e.g., geographies, regions, or clusters) over many time periods. In these settings, standard A/B testing can be inefficient or unreliable due to a limited number of units, substantial cross-unit heterogeneity, non-stationarity, and potential carryover across periods. We propose Sequentially-Rerandomized Switchback Experiments (SRSB), a new experimental design that helps mitigate these challenges. SRSB re-randomizes treatment at each time period such as to enforce balance on pre-specified prognostic variables constructed from past observations. In the absence of carryover, SRSB improves precision by leveraging temporal dependence through balancing lagged outcomes and covariates; we develop finite-sample randomization inference under a sharp null as well as asymptotic inference as the number of periods grows. We then extend SRSB to settings with first-order carryover and introduce a blocked SRSB variant that rerandomizes within strata defined by the previous treatment to form stable and comparable "stay" groups. Extensive simulations demonstrate the practical gains and robustness of SRSB relative to standard switchback designs.