Minimizing Type 2 Errors in an Experiment-Rich Regime via Optimal Resource Allocation

TL;DR

This paper develops optimal resource allocation strategies for multiple concurrent experiments to minimize Type II errors, especially when standard deviations are unknown and must be estimated from pilot data, improving detection of meaningful effects.

Contribution

It introduces new frameworks for allocating resources to minimize Type II errors, including methods for correcting pilot estimates and tractable surrogate optimization models.

Findings

Power-optimal allocation differs from MSE-based strategies.

Inflating pilot estimates improves detection power.

Surrogate-S achieves near-oracle performance in experiments.

Abstract

Randomized experiments (often known as "A/B tests") are widely used to evaluate product and service innovations. We study how to allocate limited experimentation resources across M concurrent experiments in an experiment-rich regime. Existing work on allocation has predominantly focused on minimizing the worst-case mean squared error (MSE) of estimated treatment effects, which favors experiments with larger (and typically unknown) outcome variance. While appropriate for controlling estimation accuracy, this objective does not directly capture a common managerial priority in screening stages: detecting practically meaningful treatment effects with high probability. Motivated by this, we consider the objective of minimizing the worst-case Type II error across all experiments. When the standard deviations are known, we characterize the power-optimal allocation and show that MSE-based allocations can be highly inefficient for detection, even though the two objectives align asymptotically. When the standard deviations are unknown and must be learned from pilot data, we show that a naive plug-in approach, treating pilot standard deviations as truth, can suffer substantial power loss. We propose inflating pilot estimates via correction factors and develop three optimization-based frameworks for selecting them, each reflecting a different risk criterion with distinct managerial implications. Although the resulting stochastic programs are computationally challenging at scale, we derive tractable surrogate reformulations inspired by robust optimization and establish favorable theoretical properties. We further propose Surrogate-S, a fully data-dependent and implementable procedure that computes correction factors using only pilot variance estimates and achieves near-oracle performance in numerical experiments.