Conducting A/B Experiments with a Scalable Architecture

TL;DR

This paper proposes a four-principle software architecture for scalable, domain-agnostic A/B experiments that facilitates rapid deployment, understanding, and automated analysis of experimental results.

Contribution

It introduces a principled, scalable architecture for A/B experiment software systems that addresses resource constraints and enhances experiment management.

Findings

Successfully developed and implemented the architecture in a real-world experiment.

The system supports rapid experiment deployment and automated analysis.

Enhances understanding of participant behavior and outcomes.

Abstract

A/B experiments are commonly used in research to compare the effects of changing one or more variables in two different experimental groups - a control group and a treatment group. While the benefits of using A/B experiments are widely known and accepted, there is less agreement on a principled approach to creating software infrastructure systems to assist in rapidly conducting such experiments. We propose a four-principle approach for developing a software architecture to support A/B experiments that is domain agnostic and can help alleviate some of the resource constraints currently needed to successfully implement these experiments: the software architecture (i) must retain the typical properties of A/B experiments, (ii) capture problem solving activities and outcomes, (iii) allow researchers to understand the behavior and outcomes of participants in the experiment, and (iv) must enable automated analysis. We successfully developed a software system to encapsulate these principles and implement it in a real-world A/B experiment.