A Comparison of Human and ChatGPT Classification Performance on Complex Social Media Data

TL;DR

This study evaluates GPT-4's ability to classify complex social media data compared to human annotators, highlighting its strengths and limitations in handling nuanced language.

Contribution

The paper provides a comparative analysis of GPT-4 and earlier versions on nuanced classification tasks, revealing performance gaps and influencing best practices.

Findings

GPT-4 shows moderate accuracy but struggles with nuanced language.

Including label definitions in prompts can improve GPT-4 performance.

Qualitative analysis identifies specific challenges faced by GPT-4 in classification.

Abstract

Generative artificial intelligence tools, like ChatGPT, are an increasingly utilized resource among computational social scientists. Nevertheless, there remains space for improved understanding of the performance of ChatGPT in complex tasks such as classifying and annotating datasets containing nuanced language. Method. In this paper, we measure the performance of GPT-4 on one such task and compare results to human annotators. We investigate ChatGPT versions 3.5, 4, and 4o to examine performance given rapid changes in technological advancement of large language models. We craft four prompt styles as input and evaluate precision, recall, and F1 scores. Both quantitative and qualitative evaluations of results demonstrate that while including label definitions in prompts may help performance, overall GPT-4 has difficulty classifying nuanced language. Qualitative analysis reveals four specific findings. Our results suggest the use of ChatGPT in classification tasks involving nuanced language should be conducted with prudence.