The Illusion of Learning from Observational Data: An Empirical Bayes Perspective

TL;DR

This paper proposes an empirical Bayes approach to assess and correct for biases in observational studies by using calibration experiments with known zero effects, improving causal inference.

Contribution

It introduces a method to learn the distribution of observational biases from calibration studies, enabling more accurate causal estimates from observational data.

Findings

Bias distribution can be consistently estimated with more calibration studies.

Empirical Bayes shrinkage improves causal effect estimation.

Simulation and semi-synthetic examples demonstrate method effectiveness.

Abstract

Randomized experiments have long been the gold standard for scientists seeking to learn about cause and effect. When randomized experiments are infeasible, scientists often resort to observational studies, which are widely available and often large but rely on untestable assumptions that, when violated, may result in biased estimates. Uncertainty about bias leads to a phenomenon known as the illusion of learning from observational research (Gerber, Green and Kaplan, 2004a): absent prior information about bias, observational results cannot meaningfully contribute to the estimation of a causal parameter. To shatter the illusion, we take an empirical Bayes perspective. We show that the distribution of observational biases can be learned from calibration studies-experiments that target a causal effect that is known a priori to be zero. Calibration identifies the distribution of observational bias and allows observational studies to inform the estimation of causal parameters via empirical Bayes shrinkage. We formalize the illusion phenomenon in an empirical Bayes setting and show that, with an increasing number of calibration and observation studies, both the bias distribution and the causal effect can be consistently recovered. We illustrate our method through a simulation study and a semi-synthetic application based on Ferraro and Miranda (2013)'s water-usage experiment.