Accurate Measures of Vaccination and Concerns of Vaccine Holdouts from Web Search Logs

TL;DR

This paper introduces a machine learning approach using search logs to accurately measure COVID-19 vaccine intent and hesitancy, revealing detailed regional, demographic, and temporal insights into vaccine behaviors.

Contribution

It develops a novel search-based classifier for vaccine intent, enabling real-time, granular analysis of vaccine hesitancy and concerns across different populations.

Findings

High correlation (above 0.86) with CDC vaccination rates

Holdouts are 69% more likely to click untrusted news sites

Vaccine concerns vary significantly across demographics

Abstract

To design effective vaccine policies, policymakers need detailed data about who has been vaccinated, who is holding out, and why. However, existing data in the US are insufficient: reported vaccination rates are often delayed or missing, and surveys of vaccine hesitancy are limited by high-level questions and self-report biases. Here, we show how large-scale search engine logs and machine learning can be leveraged to fill these gaps and provide novel insights about vaccine intentions and behaviors. First, we develop a vaccine intent classifier that can accurately detect when a user is seeking the COVID-19 vaccine on search. Our classifier demonstrates strong agreement with CDC vaccination rates, with correlations above 0.86, and estimates vaccine intent rates to the level of ZIP codes in real time, allowing us to pinpoint more granular trends in vaccine seeking across regions, demographics, and time. To investigate vaccine hesitancy, we use our classifier to identify two groups, vaccine early adopters and vaccine holdouts. We find that holdouts, compared to early adopters matched on covariates, are 69% more likely to click on untrusted news sites. Furthermore, we organize 25,000 vaccine-related URLs into a hierarchical ontology of vaccine concerns, and we find that holdouts are far more concerned about vaccine requirements, vaccine development and approval, and vaccine myths, and even within holdouts, concerns vary significantly across demographic groups. Finally, we explore the temporal dynamics of vaccine concerns and vaccine seeking, and find that key indicators emerge when individuals convert from holding out to preparing to accept the vaccine.