Low incidence rate of COVID-19 undermines confidence in estimation of the vaccine efficacy

TL;DR

This paper demonstrates that low COVID-19 incidence rates can lead to underestimated uncertainty in vaccine efficacy estimates, proposing new confidence intervals and sample size calculations to improve accuracy.

Contribution

It introduces a probabilistic model revealing underestimation of efficacy uncertainty at low disease risks and proposes new methods for confidence intervals and sample size determination.

Findings

Broader confidence intervals for COVID-19 vaccine efficacy at low incidence rates

Underestimation of required sample sizes in phase 3 trials

Impact of classification bias at low disease prevalence

Abstract

Knowing the true effect size of clinical interventions in randomised clinical trials is key to informing the public health policies. Vaccine efficacy is defined in terms of the relative risk or the ratio of two disease risks. However, only approximate methods are available for estimating the variance of the relative risk. In this article, we show using a probabilistic model that uncertainty in the efficacy rate could be underestimated when the disease risk is low. Factoring in the baseline rate of the disease, we estimate broader confidence intervals for the efficacy rates of the vaccines recently developed for COVID-19. We propose new confidence intervals for the relative risk. We further show that sample sizes required for phase 3 efficacy trials are routinely underestimated and propose a new method for sample size calculation where the efficacy is of interest. We also discuss the deleterious effects of classification bias which is particularly relevant at low disease prevalence.