Policy Evaluation during a Pandemic

TL;DR

This paper evaluates the effectiveness of Covid-19 policies using epidemiologically consistent identification strategies, highlighting the limitations of traditional methods and proposing alternatives for policy impact assessment.

Contribution

It compares common identification strategies against epidemic models and introduces alternative methods better suited for nonlinear pandemic data.

Findings

Unconfoundedness approaches are more suitable than difference-in-differences during a pandemic.

Traditional difference-in-differences may be biased due to nonlinear case spread.

Proposed methods effectively evaluate policy impacts on Covid-19 and economic outcomes.

Abstract

National and local governments have implemented a large number of policies in response to the Covid-19 pandemic. Evaluating the effects of these policies, both on the number of Covid-19 cases as well as on other economic outcomes is a key ingredient for policymakers to be able to determine which policies are most effective as well as the relative costs and benefits of particular policies. In this paper, we consider the relative merits of common identification strategies that exploit variation in the timing of policies across different locations by checking whether the identification strategies are compatible with leading epidemic models in the epidemiology literature. We argue that unconfoundedness type approaches, that condition on the pre-treatment "state" of the pandemic, are likely to be more useful for evaluating policies than difference-in-differences type approaches due to the highly nonlinear spread of cases during a pandemic. For difference-in-differences, we further show that a version of this problem continues to exist even when one is interested in understanding the effect of a policy on other economic outcomes when those outcomes also depend on the number of Covid-19 cases. We propose alternative approaches that are able to circumvent these issues. We apply our proposed approach to study the effect of state level shelter-in-place orders early in the pandemic.