Online Experimentation with Surrogate Metrics: Guidelines and a Case Study

TL;DR

This paper discusses how to effectively use surrogate metrics in online A/B testing to make faster decisions, ensuring trustworthy results despite imperfect predictions, with practical guidelines and a LinkedIn case study.

Contribution

It provides a framework for adjusting A/B tests with surrogate metrics and offers practical guidelines for selecting effective surrogates, illustrated by a LinkedIn case study.

Findings

Adjusted A/B testing methods improve trustworthiness of surrogate-based decisions

Guidelines help select good surrogate metrics for long-term outcomes

Case study demonstrates practical application in LinkedIn job marketplace

Abstract

A/B tests have been widely adopted across industries as the golden rule that guides decision making. However, the long-term true north metrics we ultimately want to drive through A/B test may take a long time to mature. In these situations, a surrogate metric which predicts the long-term metric is often used instead to conclude whether the treatment is effective. However, because the surrogate rarely predicts the true north perfectly, a regular A/B test based on surrogate metrics tends to have high false positive rate and the treatment variant deemed favorable from the test may not be the winning one. In this paper, we discuss how to adjust the A/B testing comparison to ensure experiment results are trustworthy. We also provide practical guidelines on the choice of good surrogate metrics. To provide a concrete example of how to leverage surrogate metrics for fast decision making, we present a case study on developing and evaluating the predicted confirmed hire surrogate metric in LinkedIn job marketplace.