Trustworthy Online Marketplace Experimentation with Budget-split Design
Min Liu, Jialiang Mao, Kang Kang

TL;DR
This paper introduces a new unbiased experimental design for online marketplaces with budget constraints, improving the validity and power of A/B testing in complex market environments.
Contribution
It proposes a novel budget-split experimental design that addresses bias and low power issues in two-sided online marketplaces, with practical system architecture and empirical validation.
Findings
The new design is unbiased under any buyer budget scenario.
It outperforms existing designs in statistical power.
Empirical results confirm improved experiment validity and efficiency.
Abstract
Online experimentation, also known as A/B testing, is the gold standard for measuring product impacts and making business decisions in the tech industry. The validity and utility of experiments, however, hinge on unbiasedness and sufficient power. In two-sided online marketplaces, both requirements are called into question. The Bernoulli randomized experiments are biased because treatment units interfere with control units through market competition and violate the "stable unit treatment value assumption"(SUTVA). The experimental power on at least one side of the market is often insufficient because of disparate sample sizes on the two sides. Despite the important of online marketplaces to the online economy and the crucial role experimentation plays in product improvement, there lacks an effective and practical solution to the bias and low power problems in marketplace experimentation.…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMobile Crowdsensing and Crowdsourcing · Innovative Microfluidic and Catalytic Techniques Innovation · Data Stream Mining Techniques
