Simulation and inference on purely observational methods of monitoring vaccine effectiveness post-deployment: none is reliable without precise information on population behaviour

TL;DR

This paper critically evaluates observational methods for monitoring vaccine effectiveness post-deployment, revealing that without precise population behavior data, these methods are unreliable and often inconclusive.

Contribution

It demonstrates through simulation that both crude and TNCC methods lack reliability without strong prior information on population behavior, emphasizing the need for RCTs or detailed behavioral data.

Findings

Bayesian confidence intervals are often too wide to be useful.

Neither crude nor TNCC methods are reliably better without strong priors.

Effective monitoring requires detailed behavioral data or RCTs.

Abstract

Two observational methods are currently being used to monitor post-deployment vaccine effectiveness: the obvious crude method comparing rate testing positive per head of vaccinated population with that rate per head of unvaccinated population; and the test-negative case control (TNCC) method. The two methods give very different results. We want to know whether either method is reliable. We assume either a homogeneous population or one partitioned into two homogeneous subsets which differ only in their not-directly-observable healthcare-seeking behaviour including probability of getting vaccinated. We first consider uniform independent priors on the probabilities of being hospitalised conditional on subset, vaccination status, and infection status. We simulate from the resulting model and observe the TNCC estimate, the crude estimate, and the Bayesian central 95% confidence interval on vaccine effectiveness represented as log ratio of odds ratios for infection with and without vaccination. With these wide open priors, even when the population is homogeneous, the Bayesian 95% confidence interval typically has a width of nearly 4 nats (55-fold), implying too much uncertainty for the data collected to be of any use in monitoring effectiveness. There do exist some tight priors under which the data is useful: some lead to TNCC being more accurate while with others the crude estimate is more accurate. Thus using only data from those spontaneously choosing to be tested, we find that neither method is reliably better than the other, and indeed that the desired information is not present in this data. We conclude that effective monitoring of vaccine effectiveness and side-effects requires either strong information on the population's behaviour, or ongoing randomised controlled trials (RCTs), rather than just choosing whichever of TNCC and crude estimate gives the result we prefer to find.