Selecting Experimental Sites for External Validity

TL;DR

This paper develops a Bayesian decision-theoretic framework for selecting experimental sites to maximize external validity of policy evidence, demonstrating significant efficiency gains over naive site selection methods.

Contribution

It introduces a novel Bayesian approach to optimize site selection for experiments, incorporating heterogeneity and structural models to improve external validity.

Findings

Optimal site selection improves policy effectiveness

Random site selection leads to efficiency losses

Using sites with largest effects is suboptimal

Abstract

Policy decisions often depend on evidence generated elsewhere. We take a Bayesian decision-theoretic approach to choosing where to experiment to optimize external validity. We frame external validity through a policy lens, developing a prior specification for the joint distribution of site-level treatment effects using a microeconometric structural model and allowing for other sources of heterogeneity. With data from South Asia, we show that, relative to basing policies on experiments in optimal sites, large efficiency losses result from instead using evidence from randomly-selected sites or, conversely, from sites with the largest expected treatment effects.