Multi-Label Classification of COVID-Tweets Using Large Language Models

TL;DR

This paper develops and compares multi-label classifiers for detecting COVID-19 vaccine concerns in tweets, finding that supervised BERT models outperform zero-shot GPT-3.5 in this task.

Contribution

The study evaluates multiple models, including BERT and GPT-3.5, for multi-label classification of vaccine-related concerns in social media posts, demonstrating the effectiveness of supervised models.

Findings

Supervised BERT achieved a macro-F1 score of 0.66.

BERT outperformed GPT-3.5 in this classification task.

The model ranked sixth in a competitive evaluation.

Abstract

Vaccination is important to minimize the risk and spread of various diseases. In recent years, vaccination has been a key step in countering the COVID-19 pandemic. However, many people are skeptical about the use of vaccines for various reasons, including the politics involved, the potential side effects of vaccines, etc. The goal in this task is to build an effective multi-label classifier to label a social media post (particularly, a tweet) according to the specific concern(s) towards vaccines as expressed by the author of the post. We tried three different models-(a) Supervised BERT-large-uncased, (b) Supervised HateXplain model, and (c) Zero-Shot GPT-3.5 Turbo model. The Supervised BERT-large-uncased model performed best in our case. We achieved a macro-F1 score of 0.66, a Jaccard similarity score of 0.66, and received the sixth rank among other submissions. Code is available at-https://github.com/anonmous1981/AISOME