Leveraging Large Language Models and Weak Supervision for Social Media data annotation: an evaluation using COVID-19 self-reported vaccination tweets

TL;DR

This paper evaluates the effectiveness of GPT-4 and weak supervision techniques in automatically annotating COVID-19 vaccine-related tweets, aiming to reduce manual effort and improve annotation efficiency for public health insights.

Contribution

It demonstrates that GPT-4 can accurately identify vaccine-related tweets using weak supervision without fine-tuning, offering a scalable annotation approach.

Findings

GPT-4 achieves high accuracy in tweet classification.

Weak supervision reduces annotation costs.

Performance comparable to human annotators.

Abstract

The COVID-19 pandemic has presented significant challenges to the healthcare industry and society as a whole. With the rapid development of COVID-19 vaccines, social media platforms have become a popular medium for discussions on vaccine-related topics. Identifying vaccine-related tweets and analyzing them can provide valuable insights for public health research-ers and policymakers. However, manual annotation of a large number of tweets is time-consuming and expensive. In this study, we evaluate the usage of Large Language Models, in this case GPT-4 (March 23 version), and weak supervision, to identify COVID-19 vaccine-related tweets, with the purpose of comparing performance against human annotators. We leveraged a manu-ally curated gold-standard dataset and used GPT-4 to provide labels without any additional fine-tuning or instructing, in a single-shot mode (no additional prompting).