Investigating Vaccine Buyer's Remorse: Post-Vaccination Decision Regret in COVID-19 Social Media Using Politically Diverse Human Annotation

TL;DR

This study analyzes COVID-19 vaccine regret on social media, creating a dataset with diverse annotations, and uses language models to quantify and understand the reasons behind vaccine buyer's remorse.

Contribution

The paper introduces a novel dataset from YouTube news comments, annotated by politically diverse individuals, and applies LLMs to analyze vaccine regret in social media discourse.

Findings

Vaccine buyer's remorse appears in less than 2% of posts.

Regret is mainly found in vaccine-skeptic communities.

First-person accounts citing health issues are the primary expression of regret.

Abstract

A significant gap exists in datasets regarding post-COVID-19 vaccination experiences, particularly ``vaccine buyer's remorse''. Understanding the prevalence and nature of vaccine regret, whether based on personal or vicarious experiences, is vital for addressing vaccine hesitancy and refining public health communication. In this paper, we curate a novel dataset from a large YouTube news corpus capturing COVID-19 vaccination experiences, and construct a benchmark subset focused on vaccine regret, annotated by a politically diverse panel to account for the subjective and often politicized nature of the topic. We utilize large language models (LLMs) to identify posts expressing vaccine regret, analyze the reasons behind this regret, and quantify its occurrence in both first and second-person accounts. This paper aims to (1) quantify the prevalence of vaccine regret; (2) identify common reasons for this sentiment; (3) analyze differences between first-person and vicarious experiences; and (4) assess potential biases introduced by different LLMs. We find that while vaccine buyer's remorse appears in only $<2\%$ of public discourse, it is disproportionately concentrated in vaccine-skeptic influencer communities and is predominantly expressed through first-person narratives citing adverse health events.