Covariance Estimation and its Application in Large-Scale Online Controlled Experiments

TL;DR

This paper introduces a new covariance estimation algorithm that reduces computational costs in large-scale online controlled experiments, enabling more efficient and flexible analysis of millions of user metrics.

Contribution

The paper presents a novel covariance estimation method that balances computational efficiency and accuracy, tailored for large-scale online experiments.

Findings

Reduces computational costs significantly in large-scale settings.

Enables more flexible trade-offs between cost and precision.

Facilitates applications like variance reduction and Bayesian optimization.

Abstract

During the last few decades, online controlled experiments (also known as A/B tests) have been adopted as a golden standard for measuring business improvements in industry. In our company, there are more than a billion users participating in thousands of experiments simultaneously, and with statistical inference and estimations conducted to thousands of online metrics in those experiments routinely, computational costs would become a large concern. In this paper we propose a novel algorithm for estimating the covariance of online metrics, which introduces more flexibility to the trade-off between computational costs and precision in covariance estimation. This covariance estimation method reduces computational cost of metric calculation in large-scale setting, which facilitates further application in both online controlled experiments and adaptive experiments scenarios like variance reduction, continuous monitoring, Bayesian optimization, etc., and it can be easily implemented in engineering practice.