A Framework for Network AB Testing

TL;DR

This paper introduces a comprehensive framework for network A/B testing that accounts for social influence among users, addressing limitations of traditional methods in social network contexts.

Contribution

It proposes a five-step framework for network A/B testing, incorporating social interactions, which is a novel approach compared to traditional independent user models.

Findings

Framework performs well in simulation studies

Addresses social influence in network A/B testing

Extends traditional methods to social network contexts

Abstract

A/B testing, also known as controlled experiment, bucket testing or splitting testing, has been widely used for evaluating a new feature, service or product in the data-driven decision processes of online websites. The goal of A/B testing is to estimate or test the difference between the treatment effects of the old and new variations. It is a well-studied two-sample comparison problem if each user's response is influenced by her treatment only. However, in many applications of A/B testing, especially those in HIVE of Yahoo and other social networks of Microsoft, Facebook, LinkedIn, Twitter and Google, users in the social networks influence their friends via underlying social interactions, and the conventional A/B testing methods fail to work. This paper considers the network A/B testing problem and provide a general framework consisting of five steps: data sampling, probabilistic model, parameter inference, computing average treatment effect and hypothesis test. The framework performs well for network A/B testing in simulation studies.