Tackling Interference Induced by Data Training Loops in A/B Tests: A Weighted Training Approach

TL;DR

This paper proposes a weighted training method to reduce interference in A/B tests caused by data training loops in recommendation systems, improving the accuracy of treatment effect estimation.

Contribution

The paper introduces a novel weighted training approach that predicts data point origins and applies weighted losses, reducing bias and variance in A/B testing under distribution shifts.

Findings

Lower bias and variance compared to existing methods

Achieves minimal variance among estimators without distribution shifts

Demonstrated effectiveness through simulation studies

Abstract

In modern recommendation systems, the standard pipeline involves training machine learning models on historical data to predict user behaviors and improve recommendations continuously. However, these data training loops can introduce interference in A/B tests, where data generated by control and treatment algorithms, potentially with different distributions, are combined. To address these challenges, we introduce a novel approach called weighted training. This approach entails training a model to predict the probability of each data point appearing in either the treatment or control data and subsequently applying weighted losses during model training. We demonstrate that this approach achieves the least variance among all estimators that do not cause shifts in the training distributions. Through simulation studies, we demonstrate the lower bias and variance of our approach compared to other methods.