Seesaw Experimentation: A/B Tests with Spillovers

TL;DR

This paper investigates how spillover effects in A/B testing can cause overall performance to decline despite improvements in specific metrics, and proposes a positive hurdle rate to mitigate these negative externalities.

Contribution

It introduces the concept of seesaw experimentation and derives an optimal hurdle rate to balance experimentation benefits with spillover mitigation.

Findings

Identifies seesaw experimentation as a challenge in A/B testing.

Proposes a positive hurdle rate to reduce negative spillovers.

Derives an optimal hurdle rate for practical implementation.

Abstract

This paper examines how spillover effects in A/B testing can impede organizational progress and develops strategies for mitigating these challenges. We identify a phenomenon termed ``seesaw experimentation'', where a firm's overall performance paradoxically deteriorates despite achieving continuous improvements in measured A/B testing metrics. Seesaw experimentation arises when successful innovations in primary metrics generate negative externalities in secondary, unmeasured dimensions. To address this problem, we propose implementing a positive hurdle rate for A/B test approval. We derive the optimal hurdle rate, offering a simple solution that preserves decentralized experimentation while mitigating negative spillovers.