The Bias-Variance Tradeoff in Long-Term Experimentation

TL;DR

This paper explores the persistent bias-variance tradeoff in long-term experimentation, emphasizing that accepting some bias can improve decision-making and system performance over time, especially as systems mature.

Contribution

It analyzes the long-term implications of bias in experiments, showing that bias can be acceptable and beneficial for optimization over time, and refines understanding of launch criteria effects.

Findings

Bias-variance tradeoff persists in long-term settings.

Accepting bias can improve long-term decision quality.

More stringent launch criteria are better as systems mature.

Abstract

As we exhaust methods that reduces variance without introducing bias, reducing variance in experiments often requires accepting some bias, using methods like winsorization or surrogate metrics. While this bias-variance tradeoff can be optimized for individual experiments, bias may accumulate over time, raising concerns for long-term optimization. We analyze whether bias is ever acceptable when it can accumulate, and show that a bias-variance tradeoff persists in long-term settings. Improving signal-to-noise remains beneficial, even if it introduces bias. This implies we should shift from thinking there is a single ``correct'', unbiased metric to thinking about how to make the best estimates and decisions when better precision can be achieved at the expense of bias. Furthermore, our model adds nuance to previous findings that suggest less stringent launch criterion leads to improved gains. We show while this is beneficial when the system is far from the optimum, more stringent launch criterion is preferable as the system matures.