Ensure A/B Test Quality at Scale with Automated Randomization Validation and Sample Ratio Mismatch Detection

TL;DR

This paper presents automated methods for ensuring A/B test quality at large scale, focusing on randomization validation with PSI and sample ratio mismatch detection to improve trustworthiness and efficiency.

Contribution

It introduces automated processes for randomization validation and sample ratio mismatch detection, enhancing large-scale experiment quality monitoring.

Findings

Automated randomization validation reduces false positives.

Sample ratio mismatch detection improves experiment trustworthiness.

Methods enable scalable and efficient experiment monitoring.

Abstract

eBay's experimentation platform runs hundreds of A/B tests on any given day. The platform integrates with the tracking infrastructure and customer experience servers, provides the sampling service for experiments, and has the responsibility to monitor the progress of each A/B test. There are many challenges especially when it is required to ensure experiment quality at the large scale. We discuss two automated test quality monitoring processes and methodologies, namely randomization validation using population stability index (PSI) and sample ratio mismatch (a.k.a. sample delta) detection using sequential analysis. The automated processes assist the experimentation platform to run high quality and trustworthy tests not only effectively on a large scale, but also efficiently by minimizing false positive monitoring alarms to experimenters.