Robust Sequential Experimental Design for A/B Testing

TL;DR

This paper develops a robust sequential experimental design framework for A/B testing that handles model misspecification, improving sample efficiency and providing theoretical error bounds.

Contribution

It introduces a unified approach applicable to both contextual bandit and dynamic settings, addressing robustness issues in experimental design.

Findings

Bounds worst-case mean squared error of treatment effect estimates.

Demonstrates effectiveness on synthetic and real-world datasets.

Addresses model misspecification in experimental design.

Abstract

Experimental design has emerged as a powerful approach for improving the sample efficiency of A/B testing, yet existing designs rely critically on correctly specified models. We study robust sequential experimental design under model misspecification and develop a unified framework that covers both contextual bandit and dynamic settings. Theoretically, we prove that our design bounds the worst-case mean squared error of the estimated treatment effect. Empirically, we demonstrate the effectiveness of the proposed approach using synthetic and real-world datasets from a leading technology company.