# Trustworthy Experimentation Under Telemetry Loss

## TL;DR

This paper addresses the challenge of telemetry data loss in online experiments, proposing a theoretical and practical framework to measure and mitigate its bias, thereby enhancing the reliability of data-driven decisions.

## Contribution

It introduces a formal analysis of telemetry loss bias and provides a general framework with solutions to measure absolute telemetry loss levels in large-scale systems.

## Key findings

- Telemetry loss can significantly bias experiment results.
- The proposed framework effectively quantifies telemetry loss impact.
- Applied at Microsoft, the framework improves experiment trustworthiness.

## Abstract

Failure to accurately measure the outcomes of an experiment can lead to bias and incorrect conclusions. Online controlled experiments (aka AB tests) are increasingly being used to make decisions to improve websites as well as mobile and desktop applications. We argue that loss of telemetry data (during upload or post-processing) can skew the results of experiments, leading to loss of statistical power and inaccurate or erroneous conclusions. By systematically investigating the causes of telemetry loss, we argue that it is not practical to entirely eliminate it. Consequently, experimentation systems need to be robust to its effects. Furthermore, we note that it is nontrivial to measure the absolute level of telemetry loss in an experimentation system. In this paper, we take a top-down approach towards solving this problem. We motivate the impact of loss qualitatively using experiments in real applications deployed at scale, and formalize the problem by presenting a theoretical breakdown of the bias introduced by loss. Based on this foundation, we present a general framework for quantitatively evaluating the impact of telemetry loss, and present two solutions to measure the absolute levels of loss. This framework is used by well-known applications at Microsoft, with millions of users and billions of sessions. These general principles can be adopted by any application to improve the overall trustworthiness of experimentation and data-driven decision making.