The Causal Roadmap and Simulations to Improve the Rigor and Reproducibility of Real-Data Applications

TL;DR

This paper advocates for using realistic simulations based on the Causal Roadmap to select and pre-specify statistical estimators, thereby enhancing the rigor and reproducibility of causal inference in real-data applications.

Contribution

It introduces a systematic simulation-based approach aligned with the Causal Roadmap to improve estimator selection and analysis plan pre-specification in causal studies.

Findings

Simulations inform better estimator choice in longitudinal studies.

Simulations help control Type-I error in cluster randomized trials.

Pre-specified analysis plans improve research reproducibility.

Abstract

The Causal Roadmap outlines a systematic approach to asking and answering questions of cause-and-effect: define the quantity of interest, evaluate needed assumptions, conduct statistical estimation, and carefully interpret results. To protect research integrity, it is essential that the algorithm for statistical estimation and inference be pre-specified prior to conducting any effectiveness analyses. However, it is often unclear which algorithm will perform optimally for the real-data application. Instead, there is a temptation to simply implement one's favorite algorithm -- recycling prior code or relying on the default settings of a computing package. Here, we call for the use of simulations that realistically reflect the application, including key characteristics such as strong confounding and dependent or missing outcomes, to objectively compare candidate estimators and facilitate full specification of the Statistical Analysis Plan. Such simulations are informed by the Causal Roadmap and conducted after data collection but prior to effect estimation. We illustrate with two worked examples. First, in an observational longitudinal study, outcome-blind simulations are used to inform nuisance parameter estimation and variance estimation for longitudinal targeted minimum loss-based estimation (TMLE). Second, in a cluster randomized trial with missing outcomes, treatment-blind simulations are used to examine Type-I error control in Two-Stage TMLE. In both examples, realistic simulations empower us to pre-specify an estimation approach that is expected to have strong finite sample performance and also yield quality-controlled computing code for the actual analysis. Together, this process helps to improve the rigor and reproducibility of our research.