How to Measure Your App: A Couple of Pitfalls and Remedies in Measuring App Performance in Online Controlled Experiments
Yuxiang Xie, Meng Xu, Evan Chow, Xiaolin Shi

TL;DR
This paper identifies pitfalls in industry-standard mobile app performance measurement during A/B testing, and proposes scalable remedies like user-level metrics and imputation, validated through simulations and real experiments at Snap.
Contribution
It uncovers two major pitfalls in measuring app performance in A/B tests and introduces scalable methods to address these issues, improving accuracy and reliability.
Findings
Identified heterogeneity and self-selection bias as key pitfalls.
Proposed scalable methods including user-level metrics and imputation.
Validated methods through extensive simulations and real-world deployment.
Abstract
Effectively measuring, understanding, and improving mobile app performance is of paramount importance for mobile app developers. Across the mobile Internet landscape, companies run online controlled experiments (A/B tests) with thousands of performance metrics in order to understand how app performance causally impacts user retention and to guard against service or app regressions that degrade user experiences. To capture certain characteristics particular to performance metrics, such as enormous observation volume and high skewness in distribution, an industry-standard practice is to construct a performance metric as a quantile over all performance events in control or treatment buckets in A/B tests. In our experience with thousands of A/B tests provided by Snap, we have discovered some pitfalls in this industry-standard way of calculating performance metrics that can lead to…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdvanced Causal Inference Techniques
