Prior-Free Sample Size Design for Test-and-Roll Experiments

TL;DR

This paper introduces a welfare-aware, prior-free method for determining optimal sample sizes in test-and-roll experiments, balancing exploration benefits against welfare costs, with a practical rule of approximately one-third of the population.

Contribution

It proposes the Worst-case Marginal Benefit rule for sample-size determination, improving over standard minimax regret by addressing exploration-exploitation tradeoffs in finite-population experiments.

Findings

WMB rule yields optimal sample size of about N/3 for Bernoulli outcomes.

The same benchmark applies exactly for Gaussian outcomes with known variance.

Provides a practical, prior-free guide for welfare-based sample-size design.

Abstract

This paper studies sample-size design for finite-population test-and-roll experiments, where a decision-maker first conducts an experiment on $m$ units and then assigns the remaining $N-m$ units to the treatment that performs better in the experiment. We consider welfare-aware sample-size choice, which involves an exploration-exploitation tradeoff: larger experiments improve the rollout decision but impose welfare losses on experimental units assigned to the inferior treatment. We show that the standard absolute minimax regret criterion can lead to implausibly small experiments by over-penalizing exploration in its worst-case objective. To address this limitation, we propose the Worst-case Marginal Benefit (WMB) rule, which compares the worst-case marginal benefit of adding one more matched pair to the experiment with the corresponding marginal exploration cost. We establish a simple rule-of-thirds benchmark. For Bernoulli outcomes, after excluding pathological cases, the WMB criterion yields the optimal sample size of $m \approx N/3$ through a Gaussian approximation. For Gaussian outcomes with a known common variance, the same benchmark arises exactly. These results provide a prior-free and practically implementable guide for welfare-based sample-size design.