‘Not finding causal effect’ is not ‘finding no causal effect’ of school closure on COVID-19

TL;DR

This paper argues that a previous study's claim of no causal effect of school closures on COVID-19 incidence is not supported by its data due to methodological limitations.

Contribution

The paper highlights flaws in the original study's methodology and shows that the data could not rule out a strong mitigating effect of school closures.

Findings

The confidence intervals from the original study included a 100% relative reduction in incidence.

Simulations showed that even large mitigating effects would not be statistically significant with the original study's design.

Abstract

In a paper recently published in
Nature Medicine, Fukumoto et al. tried to assess the government-led school closure policy during the early phase of the COVID-19 pandemic in Japan. They compared the reported incidence rates between municipalities that had and had not implemented school closure in selected periods from March–May 2020, where they matched for various potential confounders, and claimed that there was no causal effect on the incidence rates of COVID-19. However, the effective sample size (ESS) of their dataset had been substantially reduced in the process of matching due to imbalanced covariates between the treatment (i.e. with closure) and control (without closure) municipalities, which led to the wide uncertainty in the estimates. Despite the study title starting with “No causal effect of school closures”, their results are insufficient to exclude the possibility of a strong mitigating effect of school closure on incidence of COVID-19. In this replication/reanalysis study, we showed that the confidence intervals of the effect estimates from Fukumoto et al. included a 100% relative reduction in COVID-19 incidence. Simulations of a hypothetical 50% or 80% mitigating effect hardly yielded statistical significance with the same study design and sample size. We also showed that matching of variables that had large influence on propensity scores (e.g. prefecture dummy variables) may have been incomplete.

Figures