A trial emulation approach for policy evaluations with group-level longitudinal data

TL;DR

This paper advocates for a structured 'policy trial emulation' approach to evaluate group-level longitudinal policy effects, exemplified by analyzing stay-at-home orders during COVID-19.

Contribution

It introduces the concept of policy trial emulation for group-level data, adapting epidemiologic trial design principles to policy impact evaluation.

Findings

Panel methods can estimate policy effects with careful data and modeling.

Constructing separate target trials for each treatment cohort improves analysis.

Methodological challenges remain in applying these approaches effectively.

Abstract

To limit the spread of the novel coronavirus, governments across the world implemented extraordinary physical distancing policies, such as stay-at-home orders, and numerous studies aim to estimate their effects. Many statistical and econometric methods, such as difference-in-differences, leverage repeated measurements and variation in timing to estimate policy effects, including in the COVID-19 context. While these methods are less common in epidemiology, epidemiologic researchers are well accustomed to handling similar complexities in studies of individual-level interventions. "Target trial emulation" emphasizes the need to carefully design a non-experimental study in terms of inclusion and exclusion criteria, covariates, exposure definition, and outcome measurement -- and the timing of those variables. We argue that policy evaluations using group-level longitudinal ("panel") data need to take a similar careful approach to study design, which we refer to as "policy trial emulation." This is especially important when intervention timing varies across jurisdictions; the main idea is to construct target trials separately for each "treatment cohort" (states that implement the policy at the same time) and then aggregate. We present a stylized analysis of the impact of state-level stay-at-home orders on total coronavirus cases. We argue that estimates from panel methods -- with the right data and careful modeling and diagnostics -- can help add to our understanding of many policies, though doing so is often challenging.