Applying the Delta method in metric analytics: A practical guide with novel ideas
Alex Deng, Ulf Knoblich, Jiannan Lu

TL;DR
This paper provides a practical guide for applying the Delta method to metric analytics in big data contexts, addressing challenges in estimation and inference with real-world examples and highlighting novel applications.
Contribution
It introduces novel applications of the Delta method in metric analytics, especially for large-scale A/B testing, with practical insights and solutions.
Findings
Effective use of the Delta method for large-scale A/B testing metrics
Closed-form formulas enable cost-efficient inference in big data
Addressing data attributes improves estimation trustworthiness
Abstract
During the last decade, the information technology industry has adopted a data-driven culture, relying on online metrics to measure and monitor business performance. Under the setting of big data, the majority of such metrics approximately follow normal distributions, opening up potential opportunities to model them directly without extra model assumptions and solve big data problems via closed-form formulas using distributed algorithms at a fraction of the cost of simulation-based procedures like bootstrap. However, certain attributes of the metrics, such as their corresponding data generating processes and aggregation levels, pose numerous challenges for constructing trustworthy estimation and inference procedures. Motivated by four real-life examples in metric development and analytics for large-scale A/B testing, we provide a practical guide to applying the Delta method, one of the…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsDistributed Sensor Networks and Detection Algorithms · Data Stream Mining Techniques · Statistical Methods and Inference
