Comparing LLM Text Annotation Skills: A Study on Human Rights Violations in Social Media Data

TL;DR

This study evaluates the annotation capabilities of various large language models in identifying human rights violations in social media posts in Russian and Ukrainian, comparing their performance to human annotations.

Contribution

It provides a comprehensive comparison of LLMs' effectiveness in complex, multilingual annotation tasks, highlighting their strengths and limitations in sensitive domains.

Findings

LLMs show varying accuracy in identifying human rights violations.

Prompt language influences model performance.

Models exhibit distinct error patterns and cross-linguistic differences.

Abstract

In the era of increasingly sophisticated natural language processing (NLP) systems, large language models (LLMs) have demonstrated remarkable potential for diverse applications, including tasks requiring nuanced textual understanding and contextual reasoning. This study investigates the capabilities of multiple state-of-the-art LLMs - GPT-3.5, GPT-4, LLAMA3, Mistral 7B, and Claude-2 - for zero-shot and few-shot annotation of a complex textual dataset comprising social media posts in Russian and Ukrainian. Specifically, the focus is on the binary classification task of identifying references to human rights violations within the dataset. To evaluate the effectiveness of these models, their annotations are compared against a gold standard set of human double-annotated labels across 1000 samples. The analysis includes assessing annotation performance under different prompting conditions, with prompts provided in both English and Russian. Additionally, the study explores the unique patterns of errors and disagreements exhibited by each model, offering insights into their strengths, limitations, and cross-linguistic adaptability. By juxtaposing LLM outputs with human annotations, this research contributes to understanding the reliability and applicability of LLMs for sensitive, domain-specific tasks in multilingual contexts. It also sheds light on how language models handle inherently subjective and context-dependent judgments, a critical consideration for their deployment in real-world scenarios.