Learning about Treatment Effects with Prior Studies: A Bayesian Model Averaging Approach

TL;DR

This paper develops a Bayesian model averaging method for estimating treatment effects using prior studies, accounting for their validity and informativeness, and introduces a new asymptotic framework for such settings.

Contribution

It introduces a novel asymptotic framework where prior precisions grow with sample size, and models source validity via an external-validity index within Bayesian model averaging.

Findings

Biased sources are exponentially downweighted based on bias and information content.

The method concentrates on unbiased sources, achieving faster convergence.

When all sources are biased, a conservative prior ensures robustness and standard convergence.

Abstract

We establish concentration rates for estimation of treatment effects in experiments that incorporate prior sources of information -- such as past pilots, related studies, or expert assessments -- whose external validity is uncertain. Each source is modeled as a Gaussian prior with its own mean and precision, and sources are combined using Bayesian model averaging (BMA), allowing data from the new experiment to update posterior weights. To capture empirically relevant settings in which prior studies may be as informative as the current experiment, we introduce a nonstandard asymptotic framework in which prior precisions grow with the experiment's sample size. In this regime, posterior weights are governed by an external-validity index that depends jointly on a source's bias and information content: biased sources are exponentially downweighted, while unbiased sources dominate. When at least one source is unbiased, our procedure concentrates on the unbiased set and achieves faster convergence than relying on new data alone. When all sources are biased, including a deliberately conservative (diffuse) prior guarantees robustness and recovers the standard convergence rate.