Large-Scale Metric Computation in Online Controlled Experiment Platform

TL;DR

This paper presents an efficient large-scale metric computation method using bit-sliced index arithmetic for online controlled experiments, demonstrated in WeChat's platform, significantly improving performance.

Contribution

It introduces the application of bit-sliced index arithmetic for scalable metric computation in online experiment platforms, with real-world implementation and performance validation.

Findings

BSI arithmetic significantly improves computation efficiency

The approach scales well with large experiment data

Implementation in WeChat demonstrates practical effectiveness

Abstract

Online controlled experiment (also called A/B test or experiment) is the most important tool for decision-making at a wide range of data-driven companies like Microsoft, Google, Meta, etc. Metric computation is the core procedure for reaching a conclusion during an experiment. With the growth of experiments and metrics in an experiment platform, computing metrics efficiently at scale becomes a non-trivial challenge. This work shows how metric computation in WeChat experiment platform can be done efficiently using bit-sliced index (BSI) arithmetic. This approach has been implemented in a real world system and the performance results are presented, showing that the BSI arithmetic approach is very suitable for large-scale metric computation scenarios.