Measuring e-Commerce Metric Changes in Online Experiments

TL;DR

This paper investigates how dependencies between transaction metrics affect the accuracy of online experiment measurements in e-commerce, emphasizing the importance of mitigation strategies to improve decision-making.

Contribution

It provides empirical evidence on metric dependencies, their impact on measurement uncertainty, and shares practical mitigation approaches integrated into a production platform.

Findings

Dependency between transaction metrics affects measurement accuracy

Ignoring dependencies can lead to misleading experiment results

Mitigation approaches improve measurement reliability

Abstract

Digital technology organizations routinely use online experiments (e.g. A/B tests) to guide their product and business decisions. In e-commerce, we often measure changes to transaction- or item-based business metrics such as Average Basket Value (ABV), Average Basket Size (ABS), and Average Selling Price (ASP); yet it remains a common pitfall to ignore the dependency between the value/size of transactions/items during experiment design and analysis. We present empirical evidence on such dependency, its impact on measurement uncertainty, and practical implications on A/B test outcomes if left unmitigated. By making the evidence available, we hope to drive awareness of the pitfall among experimenters in e-commerce and hence encourage the adoption of established mitigation approaches. We also share lessons learned when incorporating selected mitigation approaches into our experimentation analysis platform currently in production.