Meta-experiments: Improving experimentation through experimentation

TL;DR

This paper introduces 'meta-experiments', applying A/B testing to improve the experimentation process itself, demonstrating how this approach enhances test power and benefits experimenters through self-application.

Contribution

It presents the novel concept of meta-experiments, applying A/B testing to optimize the experimentation process and showcasing practical benefits for experimenters.

Findings

Meta-experiments improve test power in A/B testing.

Applying experimentation to the process itself yields practical benefits.

Self-application ('dogfooding') enhances experimenter effectiveness.

Abstract

A/B testing is widexly used in the industry to optimize customer facing websites. Many companies employ experimentation specialists to facilitate and improve the process of A/B testing. Here, we present the application of A/B testing to this improvement effort itself, by running experiments on the experimentation process, which we call 'meta-experiments'. We discuss the challenges of this approach using the example of one of our meta-experiments, which helped experimenters to run more sufficiently powered A/B tests. We also point out the benefits of 'dog fooding' for the experimentation specialists when running their own experiments.