Pareto optimal proxy metrics

TL;DR

This paper introduces Pareto optimal proxy metrics that balance short-term sensitivity and long-term impact estimation, improving decision-making in online experiments for product development.

Contribution

It proposes a novel multi-objective optimization method for proxy metrics that enhances sensitivity while maintaining accurate long-term impact estimation.

Findings

Proxy metrics were eight times more sensitive than the north star metric.

The method increased decision velocity and quality in a large industrial recommendation system.

Proxy metrics consistently aligned with long-term impact directions.

Abstract

North star metrics and online experimentation play a central role in how technology companies improve their products. In many practical settings, however, evaluating experiments based on the north star metric directly can be difficult. The two most significant issues are 1) low sensitivity of the north star metric and 2) differences between the short-term and long-term impact on the north star metric. A common solution is to rely on proxy metrics rather than the north star in experiment evaluation and launch decisions. Existing literature on proxy metrics concentrates mainly on the estimation of the long-term impact from short-term experimental data. In this paper, instead, we focus on the trade-off between the estimation of the long-term impact and the sensitivity in the short term. In particular, we propose the Pareto optimal proxy metrics method, which simultaneously optimizes prediction accuracy and sensitivity. In addition, we give an efficient multi-objective optimization algorithm that outperforms standard methods. We applied our methodology to experiments from a large industrial recommendation system, and found proxy metrics that are eight times more sensitive than the north star and consistently moved in the same direction, increasing the velocity and the quality of the decisions to launch new features.