A Method for Measuring Network Effects of One-to-One Communication Features in Online A/B Tests

TL;DR

This paper introduces a new edge-based method to accurately measure the effects of features in one-to-one messaging A/B tests, accounting for network interference that traditional methods overlook.

Contribution

It proposes a novel network-aware analysis technique for A/B testing in messaging products, addressing the violation of SUTVA due to network effects.

Findings

Standard A/B analysis underestimates feature impact in messaging contexts.

The proposed method provides more accurate effect estimates by accounting for network interference.

Empirical results from LinkedIn messaging experiments support the model's validity.

Abstract

A/B testing is an important decision making tool in product development because can provide an accurate estimate of the average treatment effect of a new features, which allows developers to understand how the business impact of new changes to products or algorithms. However, an important assumption of A/B testing, Stable Unit Treatment Value Assumption (SUTVA), is not always a valid assumption to make, especially for products that facilitate interactions between individuals. In contexts like one-to-one messaging we should expect network interference; if an experimental manipulation is effective, behavior of the treatment group is likely to influence members in the control group by sending them messages, violating this assumption. In this paper, we propose a novel method that can be used to account for network effects when A/B testing changes to one-to-one interactions. Our method is an edge-based analysis that can be applied to standard Bernoulli randomized experiments to retrieve an average treatment effect that is not influenced by network interference. We develop a theoretical model, and methods for computing point estimates and variances of effects of interest via network-consistent permutation testing. We then apply our technique to real data from experiments conducted on the messaging product at LinkedIn. We find empirical support for our model, and evidence that the standard method of analysis for A/B tests underestimates the impact of new features in one-to-one messaging contexts.