Locally Optimal Design for A/B Testing in the Presence of Covariates and Network Connection

TL;DR

This paper develops a locally optimal design method for A/B testing on social networks, accounting for network correlations and covariates to improve treatment effect estimation.

Contribution

It introduces a new design criterion incorporating network dependence and proposes a hybrid optimization approach for locally optimal treatment assignment.

Findings

Including network dependence improves A/B test accuracy.

The proposed design is robust to parameter choices.

Synthetic and real network examples validate the approach.

Abstract

A/B test, a simple type of controlled experiment, refers to the statistical procedure of experimenting to compare two treatments applied to test subjects. For example, many IT companies frequently conduct A/B tests on their users who are connected and form social networks. Often, the users' responses could be related to the network connection. In this paper, we assume that the users, or the test subjects of the experiments, are connected on an undirected network, and the responses of two connected users are correlated. We include the treatment assignment, covariate features, and network connection in a conditional autoregressive model. Based on this model, we propose a design criterion that measures the variance of the estimated treatment effect and allocate the treatment settings to the test subjects by minimizing the criterion. Since the design criterion depends on an unknown network correlation parameter, we adopt the locally optimal design method and develop a hybrid optimization approach to obtain the optimal design. Through synthetic and real social network examples, we demonstrate the value of including network dependence in designing A/B experiments and validate that the proposed locally optimal design is robust to the choices of parameters.