ChatGPT-4 Outperforms Experts and Crowd Workers in Annotating Political Twitter Messages with Zero-Shot Learning

TL;DR

This study demonstrates that ChatGPT-4 surpasses both experts and crowd workers in accurately classifying political Twitter messages, showing potential for large-scale social science research using LLMs.

Contribution

It provides empirical evidence that ChatGPT-4 outperforms human annotators in political message classification, highlighting its reliability and bias reduction in social science applications.

Findings

ChatGPT-4 achieves higher accuracy than humans.

ChatGPT-4 exhibits lower bias in classification.

LLMs can perform complex reasoning on social media data.

Abstract

This paper assesses the accuracy, reliability and bias of the Large Language Model (LLM) ChatGPT-4 on the text analysis task of classifying the political affiliation of a Twitter poster based on the content of a tweet. The LLM is compared to manual annotation by both expert classifiers and crowd workers, generally considered the gold standard for such tasks. We use Twitter messages from United States politicians during the 2020 election, providing a ground truth against which to measure accuracy. The paper finds that ChatGPT-4 has achieves higher accuracy, higher reliability, and equal or lower bias than the human classifiers. The LLM is able to correctly annotate messages that require reasoning on the basis of contextual knowledge, and inferences around the author's intentions - traditionally seen as uniquely human abilities. These findings suggest that LLM will have substantial impact on the use of textual data in the social sciences, by enabling interpretive research at a scale.