Advancing Annotation of Stance in Social Media Posts: A Comparative Analysis of Large Language Models and Crowd Sourcing

TL;DR

This study evaluates the effectiveness of eight large language models in annotating stance in social media posts, comparing their performance to human crowd-sourced judgments and identifying conditions affecting their accuracy.

Contribution

It provides a comprehensive benchmark of LLMs against human annotations for stance detection and analyzes factors influencing LLMs' agreement with humans.

Findings

LLMs perform well when stance expressions are explicit.

Disagreements often occur when humans also struggle to reach consensus.

Explicitness of stance significantly impacts LLM accuracy.

Abstract

In the rapidly evolving landscape of Natural Language Processing (NLP), the use of Large Language Models (LLMs) for automated text annotation in social media posts has garnered significant interest. Despite the impressive innovations in developing LLMs like ChatGPT, their efficacy, and accuracy as annotation tools are not well understood. In this paper, we analyze the performance of eight open-source and proprietary LLMs for annotating the stance expressed in social media posts, benchmarking their performance against human annotators' (i.e., crowd-sourced) judgments. Additionally, we investigate the conditions under which LLMs are likely to disagree with human judgment. A significant finding of our study is that the explicitness of text expressing a stance plays a critical role in how faithfully LLMs' stance judgments match humans'. We argue that LLMs perform well when human annotators do, and when LLMs fail, it often corresponds to situations in which human annotators struggle to reach an agreement. We conclude with recommendations for a comprehensive approach that combines the precision of human expertise with the scalability of LLM predictions. This study highlights the importance of improving the accuracy and comprehensiveness of automated stance detection, aiming to advance these technologies for more efficient and unbiased analysis of social media.