Fair Effect Attribution in Parallel Online Experiments

TL;DR

This paper introduces a fair method to measure and attribute the combined effects of simultaneous online experiments, using Shapley values and causal inference to ensure accurate impact assessment.

Contribution

It proposes a novel approach combining Shapley value-based cost sharing with causal inference to disentangle and fairly attribute interaction effects in parallel online experiments.

Findings

Effective in real-world online experiment settings

Accurately disentangles interaction effects

Provides fair attribution of impacts among experiments

Abstract

A/B tests serve the purpose of reliably identifying the effect of changes introduced in online services. It is common for online platforms to run a large number of simultaneous experiments by splitting incoming user traffic randomly in treatment and control groups. Despite a perfect randomization between different groups, simultaneous experiments can interact with each other and create a negative impact on average population outcomes such as engagement metrics. These are measured globally and monitored to protect overall user experience. Therefore, it is crucial to measure these interaction effects and attribute their overall impact in a fair way to the respective experimenters. We suggest an approach to measure and disentangle the effect of simultaneous experiments by providing a cost sharing approach based on Shapley values. We also provide a counterfactual perspective, that predicts shared impact based on conditional average treatment effects making use of causal inference techniques. We illustrate our approach in real world and synthetic data experiments.