Designing Experiments Toward Shrinkage Estimation

TL;DR

This paper proposes a novel experimental design framework that integrates observational data with randomized controlled trials using Empirical Bayes shrinkage, optimizing unit allocation to improve causal inference accuracy.

Contribution

It introduces algorithms for designing RCTs that leverage observational data through shrinkage estimators, including strategies to handle unmeasured confounding and sensitivity analysis.

Findings

Shrinkage estimator risk can be efficiently computed via numerical methods.

Optimized designs outperform traditional methods in simulation studies.

Incorporating observational data improves causal estimate accuracy.

Abstract

We consider how increasingly available observational data can be used to improve the design of randomized controlled trials (RCTs). We seek to design a prospective RCT, with the intent of using an Empirical Bayes estimator to shrink the causal estimates from our trial toward causal estimates obtained from an observational study. We ask: how might we design the experiment to better complement the observational study in this setting? We propose using an estimator that shrinks each component of the RCT causal estimator toward its observational counterpart by a factor proportional to its variance. First, we show that the risk of this estimator can be computed efficiently via numerical integration. We then propose algorithms for determining the best allocation of units to strata (the best "design"). We consider three options: Neyman allocation; a "naive" design assuming no unmeasured confounding in the observational study; and a "defensive" design accounting for the imperfect parameter estimates we would obtain from the observational study with unmeasured confounding. We also incorporate results from sensitivity analysis to establish guardrails on the designs, so that our experiment could be reasonably analyzed with and without shrinkage. We demonstrate the superiority of these experimental designs with a simulation study involving causal inference on a rare, binary outcome.