Unraveling the Interplay between Carryover Effects and Reward Autocorrelations in Switchback Experiments

TL;DR

This paper analyzes how carryover effects and reward autocorrelations influence switchback experiment designs in A/B testing, providing guidelines for practitioners based on a comprehensive comparison of estimators in Markovian environments.

Contribution

It offers a novel, estimator-agnostic analysis of switchback design effectiveness considering carryover and autocorrelation effects, with practical guidelines for experiment design.

Findings

Design effectiveness depends on carryover effect size.

Autocorrelations among reward errors significantly impact results.

Findings are applicable across various RL estimators.

Abstract

A/B testing has become the gold standard for policy evaluation in modern technological industries. Motivated by the widespread use of switchback experiments in A/B testing, this paper conducts a comprehensive comparative analysis of various switchback designs in Markovian environments. Unlike many existing works which derive the optimal design based on specific and relatively simple estimators, our analysis covers a range of state-of-the-art estimators developed in the reinforcement learning (RL) literature. It reveals that the effectiveness of different switchback designs depends crucially on (i) the size of the carryover effect and (ii) the auto-correlations among reward errors over time. Meanwhile, these findings are estimator-agnostic, i.e., they apply to most RL estimators. Based on these insights, we provide a workflow to offer guidelines for practitioners on designing switchback experiments in A/B testing.