For objective causal inference, design trumps analysis

TL;DR

This paper emphasizes that careful design of observational studies, mimicking randomized experiments without outcome data, is crucial for objective causal inference, highlighting methods like propensity scores and principal stratification.

Contribution

It advocates for designing observational studies to closely resemble randomized experiments, using potential outcomes and covariate overlap considerations, to improve causal inference objectivity.

Findings

Designing observational studies enhances causal inference validity.

Propensity score analysis helps identify inadequate data sets.

Principal stratification aids in adjusting study design when necessary.

Abstract

For obtaining causal inferences that are objective, and therefore have the best chance of revealing scientific truths, carefully designed and executed randomized experiments are generally considered to be the gold standard. Observational studies, in contrast, are generally fraught with problems that compromise any claim for objectivity of the resulting causal inferences. The thesis here is that observational studies have to be carefully designed to approximate randomized experiments, in particular, without examining any final outcome data. Often a candidate data set will have to be rejected as inadequate because of lack of data on key covariates, or because of lack of overlap in the distributions of key covariates between treatment and control groups, often revealed by careful propensity score analyses. Sometimes the template for the approximating randomized experiment will have to be altered, and the use of principal stratification can be helpful in doing this. These issues are discussed and illustrated using the framework of potential outcomes to define causal effects, which greatly clarifies critical issues.