Evaluating Large Language Models for Health-Related Text Classification Tasks with Public Social Media Data

TL;DR

This study evaluates large language models for health-related social media text classification, demonstrating that data augmentation with LLMs enhances lightweight supervised models, while LLMs excel as zero-shot classifiers in reducing human annotation effort.

Contribution

It introduces a novel approach of using LLMs for data augmentation and zero-shot classification in health-related social media NLP tasks, outperforming traditional methods.

Findings

Data augmentation with GPT-4 improves model performance.

Supervised models outperform LLMs in zero-shot settings.

LLMs as zero-shot classifiers help reduce false negatives.

Abstract

Large language models (LLMs) have demonstrated remarkable success in NLP tasks. However, there is a paucity of studies that attempt to evaluate their performances on social media-based health-related natural language processing tasks, which have traditionally been difficult to achieve high scores in. We benchmarked one supervised classic machine learning model based on Support Vector Machines (SVMs), three supervised pretrained language models (PLMs) based on RoBERTa, BERTweet, and SocBERT, and two LLM based classifiers (GPT3.5 and GPT4), across 6 text classification tasks. We developed three approaches for leveraging LLMs for text classification: employing LLMs as zero-shot classifiers, us-ing LLMs as annotators to annotate training data for supervised classifiers, and utilizing LLMs with few-shot examples for augmentation of manually annotated data. Our comprehensive experiments demonstrate that employ-ing data augmentation using LLMs (GPT-4) with relatively small human-annotated data to train lightweight supervised classification models achieves superior results compared to training with human-annotated data alone. Supervised learners also outperform GPT-4 and GPT-3.5 in zero-shot settings. By leveraging this data augmentation strategy, we can harness the power of LLMs to develop smaller, more effective domain-specific NLP models. LLM-annotated data without human guidance for training light-weight supervised classification models is an ineffective strategy. However, LLM, as a zero-shot classifier, shows promise in excluding false negatives and potentially reducing the human effort required for data annotation. Future investigations are imperative to explore optimal training data sizes and the optimal amounts of augmented data.