Analysis of Potential Biases and Validity of Studies Using Multiverse Approaches to Assess the Impacts of Government Responses to Epidemics

TL;DR

This paper critically examines the biases and validity issues in multiverse analyses of government responses to epidemics, highlighting potential pitfalls and proposing improved hypothesis-driven methods for causal inference.

Contribution

It identifies key biases in current multiverse approaches and introduces an alternative hypothesis-driven framework to improve causal analysis of epidemic policy impacts.

Findings

Multiverse analyses can mislead when most specifications are causally invalid.

Counterexamples show multiverse approaches may suggest no effect when effects exist.

Proposes hypothesis-driven models accounting for infectiousness and policy interactions.

Abstract

We analyze the methodological approach and validity of interpretation of using national-level time-series regression analyses relating epidemic outcomes to policies that estimate many models involving permutations of analytic choices (i.e., a "multiverse" approach). Specifically, we evaluate the possible biases and pitfalls of interpretation of a multiverse approach to the context of government responses to epidemics using the example of COVID-19 and a recently published peer-reviewed paper by Bendavid and Patel (2024) along with the subsequent commentary that the two authors published discussing and interpreting the implications of their work. While we identify multiple potential errors and sources of biases in how the specific analyses were undertaken that are also relevant for other studies employing similar approaches, our most important finding involves constructing a counterexample showing that causal model specification-agnostic multiverse analyses can be incorrectly used to suggest that no consistent effect can be discovered in data especially in cases where most specifications estimated with the data are far from causally valid. Finally, we suggest an alternative approach involving hypothesis-drive specifications that explicitly account for infectiousness across jurisdictions in the analysis as well as the interconnected ways that policies and behavioral responses may evolve within and across these jurisdictions.