Equivalence Test in Multi-dimensional Space with Applications in A/B Testing

TL;DR

This paper introduces a non-parametric statistical framework for validating the randomness of data splits in multi-dimensional A/B testing, ensuring the integrity of online experiments and other applications.

Contribution

It proposes a novel randomized chi-square test for multi-dimensional data, compares it with existing methods, and demonstrates its effectiveness on real and simulated datasets.

Findings

All three methods show promising power in detecting distribution differences.

The proposed tests are applicable to both categorical and continuous data.

The methodology is validated on datasets from Adobe Experience Cloud.

Abstract

In this paper, we provide a statistical testing framework to check whether a random sample splitting in a multi-dimensional space is carried out in a valid way, which could be directly applied to A/B testing and multivariate testing to ensure the online traffic split is truly random with respect to the covariates. We believe this is an important step of quality control that is missing in many real world online experiments. Here, we propose a randomized chi-square test method, compared with propensity score and distance components (DISCO) test methods, to test the hypothesis that the post-split categorical data sets have the same multi-dimensional distribution. The methods can be easily generalized to continuous data. We also propose a resampling procedure to adjust for multiplicity which in practice often has higher power than some existing method such as Holm's procedure. We try the three methods on both simulated and real data sets from Adobe Experience Cloud and show that each method has its own advantage while all of them establish promising power. To our knowledge, we are among the first ones to formulate the validity of A/B testing into a post-experiments statistical testing problem. Our methodology is non-parametric and requires minimum assumption on the data, so it can also have a wide range of application in other areas such as clinical trials, medicine, and recommendation system where random data splitting is needed.