A Comparison of Methods for Adaptive Experimentation

TL;DR

This paper compares three adaptive experimentation methods through simulation, evaluating their effectiveness in social welfare and estimation accuracy, and introduces hybrid loss measures to identify optimal strategies for multiple experimental goals.

Contribution

It provides a comprehensive simulation-based comparison of Thompson sampling, Tempered Thompson sampling, and Exploration sampling, including novel hybrid loss measures for multi-objective optimization.

Findings

Performance of Thompson sampling varies with the number of experimental waves.

Tempered Thompson sampling balances multiple experimental aims effectively.

Exploration sampling generally performs similarly to random assignment.

Abstract

We use a simulation study to compare three methods for adaptive experimentation: Thompson sampling, Tempered Thompson sampling, and Exploration sampling. We gauge the performance of each in terms of social welfare and estimation accuracy, and as a function of the number of experimental waves. We further construct a set of novel "hybrid" loss measures to identify which methods are optimal for researchers pursuing a combination of experimental aims. Our main results are: 1) the relative performance of Thompson sampling depends on the number of experimental waves, 2) Tempered Thompson sampling uniquely distributes losses across multiple experimental aims, and 3) in most cases, Exploration sampling performs similarly to random assignment.