Detecting the Presence of COVID-19 Vaccination Hesitancy from South African Twitter Data Using Machine Learning

TL;DR

This study analyzes South African Twitter data to detect COVID-19 vaccine hesitancy using machine learning, finding that transformer-based models like BERT and RoBERTa outperform traditional methods in sentiment classification.

Contribution

It introduces a novel approach combining sentiment analysis with transformer models on South African social media data to identify vaccine hesitancy.

Findings

BERT and RoBERTa achieved F1-scores of 60% and 61%.

Transformer models outperform traditional ML models in this context.

Topic modeling on misclassified tweets provides insights for future improvements.

Abstract

Very few social media studies have been done on South African user-generated content during the COVID-19 pandemic and even fewer using hand-labelling over automated methods. Vaccination is a major tool in the fight against the pandemic, but vaccine hesitancy jeopardizes any public health effort. In this study, sentiment analysis on South African tweets related to vaccine hesitancy was performed, with the aim of training AI-mediated classification models and assessing their reliability in categorizing UGC. A dataset of 30000 tweets from South Africa were extracted and hand-labelled into one of three sentiment classes: positive, negative, neutral. The machine learning models used were LSTM, bi-LSTM, SVM, BERT-base-cased and the RoBERTa-base models, whereby their hyperparameters were carefully chosen and tuned using the WandB platform. We used two different approaches when we pre-processed our data for comparison: one was semantics-based, while the other was corpus-based. The pre-processing of the tweets in our dataset was performed using both methods, respectively. All models were found to have low F1-scores within a range of 45$\%$-55$\%$, except for BERT and RoBERTa which both achieved significantly better measures with overall F1-scores of 60$\%$ and 61$\%$, respectively. Topic modelling using an LDA was performed on the miss-classified tweets of the RoBERTa model to gain insight on how to further improve model accuracy.