Using Prior Studies to Design Experiments: An Empirical Bayes Approach

TL;DR

This paper introduces an empirical Bayes framework that uses prior study data to inform experimental design, improving efficiency and decision-making in new experiments, with theoretical guarantees and practical applications.

Contribution

It develops a novel empirical Bayes method for experimental design that leverages prior studies, connecting Bayesian meta-analysis with experimental planning.

Findings

Achieves oracle-optimal performance as prior studies increase

Characterizes the rate of regret reduction with more prior data

Demonstrates practical applications in clinical and educational experiments

Abstract

We develop an empirical Bayes framework for experimental design that leverages information from prior related studies. When a researcher has access to estimates from previous studies on similar parameters, they can use empirical Bayes to estimate an informative prior over the parameter of interest in the new study. We show how this prior can be incorporated into a decision-theoretic experimental design framework to choose optimal design. The approach is illustrated via propensity score designs in stratified randomized experiments. Our theoretical results show that the empirical Bayes design achieves oracle-optimal performance as the number of prior studies grows, and characterize the rate at which regret vanishes. To illustrate the approach, we present two empirical applications--oncology drug trials and the Tennessee Project STAR experiment. Our framework connects the Bayesian meta-analysis literature to experimental design and provides practical guidance for researchers seeking to design more efficient experiments.