Seller-Side Experiments under Interference Induced by Feedback Loops in Two-Sided Platforms

TL;DR

This paper examines how feedback loops in two-sided platforms affect seller-side experiments, revealing that such loops can distort treatment effect estimates and proposing a framework to analyze and mitigate this interference.

Contribution

It introduces a mathematical framework to analyze feedback loop interference and empirically evaluates the counterfactual interleaving design in real-world settings.

Findings

Feedback loops can lead to misleading treatment effect estimates.

Counterfactual interleaving can mitigate some interference effects.

Theoretical analysis quantifies the impact of feedback-induced interference.

Abstract

Two-sided platforms are central to modern commerce and content sharing and often utilize A/B testing for developing new features. While user-side experiments are common, seller-side experiments become crucial for specific interventions and metrics. This paper investigates the effects of interference caused by feedback loops on seller-side experiments in two-sided platforms, with a particular focus on the counterfactual interleaving design, proposed in \citet{ha2020counterfactual,nandy2021b}. These feedback loops, often generated by pacing algorithms, cause outcomes from earlier sessions to influence subsequent ones. This paper contributes by creating a mathematical framework to analyze this interference, theoretically estimating its impact, and conducting empirical evaluations of the counterfactual interleaving design in real-world scenarios. Our research shows that feedback loops can result in misleading conclusions about the treatment effects.