Exploring the Capability of ChatGPT to Reproduce Human Labels for Social Computing Tasks (Extended Version)

TL;DR

This study evaluates ChatGPT's ability to annotate social computing datasets, revealing promising performance with an average F1-score of 72%, and introduces GPT-Rater to predict annotation success for specific tasks.

Contribution

The paper demonstrates ChatGPT's potential for data annotation in social computing and introduces GPT-Rater, a tool to predict annotation quality for different datasets.

Findings

ChatGPT achieves an average F1-score of 72% across seven datasets.

Performance varies significantly across different labels.

GPT-Rater accurately predicts ChatGPT's annotation performance, especially in clickbait detection.

Abstract

Harnessing the potential of large language models (LLMs) like ChatGPT can help address social challenges through inclusive, ethical, and sustainable means. In this paper, we investigate the extent to which ChatGPT can annotate data for social computing tasks, aiming to reduce the complexity and cost of undertaking web research. To evaluate ChatGPT's potential, we re-annotate seven datasets using ChatGPT, covering topics related to pressing social issues like COVID-19 misinformation, social bot deception, cyberbully, clickbait news, and the Russo-Ukrainian War. Our findings demonstrate that ChatGPT exhibits promise in handling these data annotation tasks, albeit with some challenges. Across the seven datasets, ChatGPT achieves an average annotation F1-score of 72.00%. Its performance excels in clickbait news annotation, correctly labeling 89.66% of the data. However, we also observe significant variations in performance across individual labels. Our study reveals predictable patterns in ChatGPT's annotation performance. Thus, we propose GPT-Rater, a tool to predict if ChatGPT can correctly label data for a given annotation task. Researchers can use this to identify where ChatGPT might be suitable for their annotation requirements. We show that GPT-Rater effectively predicts ChatGPT's performance. It performs best on a clickbait headlines dataset by achieving an average F1-score of 95.00%. We believe that this research opens new avenues for analysis and can reduce barriers to engaging in social computing research.