The Impact of Annotator Personas on LLM Behavior Across the Perspectivism Spectrum

TL;DR

This paper investigates how Large Language Models (LLMs) can annotate hate speech considering different annotator personas, revealing their tendency to aggregate perspectives and the conditions under which they match human annotations.

Contribution

It introduces a framework for LLM-based annotator modeling across the perspectivism spectrum and compares its performance to traditional methods and human annotations.

Findings

LLMs selectively use demographic attributes from personas.

LLMs tend to aggregate perspectives, especially under weak perspectivism.

Performance of LLM annotations approaches human levels in strong perspectivism datasets.

Abstract

In this work, we explore the capability of Large Language Models (LLMs) to annotate hate speech and abusiveness while considering predefined annotator personas within the strong-to-weak data perspectivism spectra. We evaluated LLM-generated annotations against existing annotator modeling techniques for perspective modeling. Our findings show that LLMs selectively use demographic attributes from the personas. We identified prototypical annotators, with persona features that show varying degrees of alignment with the original human annotators. Within the data perspectivism paradigm, annotator modeling techniques that do not explicitly rely on annotator information performed better under weak data perspectivism compared to both strong data perspectivism and human annotations, suggesting LLM-generated views tend towards aggregation despite subjective prompting. However, for more personalized datasets tailored to strong perspectivism, the performance of LLM annotator modeling approached, but did not exceed, human annotators.