ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks

TL;DR

This study demonstrates that ChatGPT surpasses crowd-workers and trained annotators in accuracy, agreement, and cost-efficiency for various NLP text annotation tasks, highlighting its potential to revolutionize data labeling in NLP.

Contribution

The paper provides empirical evidence that ChatGPT outperforms crowd-workers and trained annotators in multiple annotation tasks, with higher accuracy, agreement, and lower costs.

Findings

ChatGPT achieves higher zero-shot accuracy than crowd-workers in four out of five tasks.

ChatGPT's intercoder agreement exceeds both crowd-workers and trained annotators across all tasks.

Cost per annotation with ChatGPT is less than $0.003, significantly cheaper than MTurk.

Abstract

Many NLP applications require manual data annotations for a variety of tasks, notably to train classifiers or evaluate the performance of unsupervised models. Depending on the size and degree of complexity, the tasks may be conducted by crowd-workers on platforms such as MTurk as well as trained annotators, such as research assistants. Using a sample of 2,382 tweets, we demonstrate that ChatGPT outperforms crowd-workers for several annotation tasks, including relevance, stance, topics, and frames detection. Specifically, the zero-shot accuracy of ChatGPT exceeds that of crowd-workers for four out of five tasks, while ChatGPT's intercoder agreement exceeds that of both crowd-workers and trained annotators for all tasks. Moreover, the per-annotation cost of ChatGPT is less than $0.003 -- about twenty times cheaper than MTurk. These results show the potential of large language models to drastically increase the efficiency of text classification.