Applying the Delta method in metric analytics: A practical guide with novel ideas

TL;DR

This paper provides a practical guide for applying the Delta method to metric analytics in big data contexts, addressing challenges in estimation and inference with real-world examples and highlighting novel applications.

Contribution

It introduces novel applications of the Delta method in metric analytics, especially for large-scale A/B testing, with practical insights and solutions.

Findings

Effective use of the Delta method for large-scale A/B testing metrics

Closed-form formulas enable cost-efficient inference in big data

Addressing data attributes improves estimation trustworthiness

Abstract

During the last decade, the information technology industry has adopted a data-driven culture, relying on online metrics to measure and monitor business performance. Under the setting of big data, the majority of such metrics approximately follow normal distributions, opening up potential opportunities to model them directly without extra model assumptions and solve big data problems via closed-form formulas using distributed algorithms at a fraction of the cost of simulation-based procedures like bootstrap. However, certain attributes of the metrics, such as their corresponding data generating processes and aggregation levels, pose numerous challenges for constructing trustworthy estimation and inference procedures. Motivated by four real-life examples in metric development and analytics for large-scale A/B testing, we provide a practical guide to applying the Delta method, one of the most important tools from the classic statistics literature, to address the aforementioned challenges. We emphasize the central role of the Delta method in metric analytics by highlighting both its classic and novel applications.