Just Ramp-up: Unleash the Potential of Regression-based Estimator for A/B Tests under Network Interference

TL;DR

This paper proposes a regression-based estimator leveraging multiple sequential experiments to improve causal inference under network interference, demonstrating bias reduction and enhanced performance through data merging, especially with nonlinear interference.

Contribution

It introduces a novel approach of merging data from multiple experiments to improve GATE estimation under network interference, including analysis of bias-variance tradeoff and neural network-based estimators.

Findings

Bias reduction is achieved by merging experiments with different treatment proportions.

Merging more than two steps is unnecessary under linear interference but beneficial with nonlinear interference.

Simulation results show significant performance improvements with the proposed methods.

Abstract

Recent research in causal inference under network interference has explored various experimental designs and estimation techniques to address this issue. However, existing methods, which typically rely on single experiments, often reach a performance bottleneck and face limitations in handling diverse interference structures. In contrast, we propose leveraging multiple experiments to overcome these limitations. In industry, the use of sequential experiments, often known as the ramp-up process, where traffic to the treatment gradually increases, is common due to operational needs like risk management and cost control. Our approach shifts the focus from operational aspects to the statistical advantages of merging data from multiple experiments. By combining data from sequentially conducted experiments, we aim to estimate the global average treatment effect more effectively. In this paper, we begin by analyzing the bias and variance of the linear regression estimator for GATE under general linear network interference. We demonstrate that bias plays a dominant role in the bias-variance tradeoff and highlight the intrinsic bias reduction achieved by merging data from experiments with strictly different treatment proportions. Herein the improvement introduced by merging two steps of experimental data is essential. In addition, we show that merging more steps of experimental data is unnecessary under general linear interference, while it can become beneficial when nonlinear interference occurs. Furthermore, we look into a more advanced estimator based on graph neural networks. Through extensive simulation studies, we show that the regression-based estimator benefits remarkably from training on merged experiment data, achieving outstanding statistical performance.