Dealing With Ratio Metrics in A/B Testing at the Presence of Intra-User Correlation and Segments

TL;DR

This paper introduces a correlation-adjusted mean estimator for ratio metrics in A/B testing, effectively accounting for intra-user correlation and segments to improve estimation accuracy and statistical power.

Contribution

It proposes new methods to estimate intra-user correlation and a correlation-adjusted mean estimator that outperforms traditional estimators in variance reduction and bias correction.

Findings

Correlation-adjusted mean reduces variance compared to naive and normalized means.

Accurate correlation estimation improves the power of A/B tests.

Application to eBay data demonstrates practical effectiveness.

Abstract

We study ratio metrics in A/B testing at the presence of correlation among observations coming from the same user and provides practical guidance especially when two metrics contradict each other. We propose new estimating methods to quantitatively measure the intra-user correlation (within segments). With the accurately estimated correlation, a uniformly minimum-variance unbiased estimator of the population mean, called correlation-adjusted mean, is proposed to account for such correlation structure. It is proved theoretically and numerically better than the other two unbiased estimators, naive mean and normalized mean (averaging within users first and then across users). The correlation-adjusted mean method is unbiased and has reduced variance so it gains additional power. Several simulation studies are designed to show the estimation accuracy of the correlation structure, effectiveness in reducing variance, and capability of obtaining more power. An application to the eBay data is conducted to conclude this paper.