Sensitivity Analysis when Generalizing Causal Effects from Multiple Studies to a Target Population: Motivation from the ECHO Program

TL;DR

This paper develops a flexible sensitivity analysis framework to evaluate the robustness of causal effect generalizations across multiple studies, addressing unobserved effect modifiers without relying on strict assumptions.

Contribution

It introduces an interpretable, assumption-light sensitivity analysis method for multi-study generalizability, including hypothesis testing for violations, demonstrated through simulations and real ECHO data.

Findings

High power in detecting violations of generalizability assumptions

Effective application to ECHO study data on secondhand smoke and birth weight

Framework accommodates various study designs without distributional assumptions

Abstract

Unobserved effect modifiers can induce bias when generalizing causal effect estimates to target populations. In this work, we extend a sensitivity analysis framework assessing the robustness of study results to unobserved effect modification that adapts to various generalizability scenarios, including multiple (conditionally) randomized trials, observational studies, or combinations thereof. This framework is interpretable and does not rely on distributional or functional assumptions about unknown parameters. We demonstrate how to leverage the multi-study setting to detect violation of the generalizability assumption through hypothesis testing, showing with simulations that the proposed test achieves high power under real-world sample sizes. Finally, we apply our sensitivity analysis framework to analyze the generalized effect estimate of secondhand smoke exposure on birth weight using cohort sites from the Environmental influences on Child Health Outcomes (ECHO) study.