Some Statistical and Data Challenges When Building Early-Stage Digital Experimentation and Measurement Capabilities

TL;DR

This paper discusses the statistical and data challenges in building digital experimentation and measurement capabilities, offering novel approaches, case studies, and evaluation frameworks to enhance data-driven decision-making in organizations.

Contribution

It introduces new methods for addressing statistical challenges in DEM, including a business case model, detailed case studies, and a framework for comparing experiment designs.

Findings

Quantified benefits and risks of DEM capabilities

Developed a ranking model under uncertainty for business cases

Created an evaluation framework for experiment design efficiency

Abstract

Digital experimentation and measurement (DEM) capabilities -- the knowledge and tools necessary to run experiments with digital products, services, or experiences and measure their impact -- are fast becoming part of the standard toolkit of digital/data-driven organisations in guiding business decisions. Many large technology companies report having mature DEM capabilities, and several businesses have been established purely to manage experiments for others. Given the growing evidence that data-driven organisations tend to outperform their non-data-driven counterparts, there has never been a greater need for organisations to build/acquire DEM capabilities to thrive in the current digital era. This thesis presents several novel approaches to statistical and data challenges for organisations building DEM capabilities. We focus on the fundamentals associated with building DEM capabilities, which lead to a richer understanding of the underlying assumptions and thus enable us to develop more appropriate capabilities. We address why one should engage in DEM by quantifying the benefits and risks of acquiring DEM capabilities. This is done using a ranking under lower uncertainty model, enabling one to construct a business case. We also examine what ingredients are necessary to run digital experiments. In addition to clarifying the existing literature around statistical tests, datasets, and methods in experimental design and causal inference, we construct an additional dataset and detailed case studies on applying state-of-the-art methods. Finally, we investigate when a digital experiment design would outperform another, leading to an evaluation framework that compares competing designs' data efficiency.