Designing Time Series Experiments in A/B Testing with Transformer Reinforcement Learning

TL;DR

This paper introduces a transformer reinforcement learning method for time series A/B testing, enabling better treatment allocation by fully leveraging historical data and directly optimizing the treatment effect estimation accuracy.

Contribution

It presents a novel transformer RL approach that overcomes limitations of existing designs by conditioning on full history and directly optimizing the mean squared error.

Findings

Outperforms existing designs on synthetic data

Effective in real-world ridesharing dataset

Demonstrates improved treatment effect estimation

Abstract

A/B testing has become a gold standard for modern technological companies to conduct policy evaluation. Yet, its application to time series experiments, where policies are sequentially assigned over time, remains challenging. Existing designs suffer from two limitations: (i) they do not fully leverage the entire history for treatment allocation; (ii) they rely on strong assumptions to approximate the objective function (e.g., the mean squared error of the estimated treatment effect) for optimizing the design. We first establish an impossibility theorem showing that failure to condition on the full history leads to suboptimal designs, due to the dynamic dependencies in time series experiments. To address both limitations simultaneously, we next propose a transformer reinforcement learning (RL) approach which leverages transformers to condition allocation on the entire history and employs RL to directly optimize the MSE without relying on restrictive assumptions. Empirical evaluations on synthetic data, a publicly available dispatch simulator, and a real-world ridesharing dataset demonstrate that our proposal consistently outperforms existing designs.