Large Language Models as Automatic Annotators and Annotation Adjudicators for Fine-Grained Opinion Analysis

TL;DR

This paper investigates the use of large language models as automatic tools for annotating and adjudicating fine-grained opinion analysis in text, aiming to reduce human effort and cost in dataset creation.

Contribution

It introduces a declarative annotation pipeline and a novel adjudication methodology for LLMs, demonstrating their effectiveness in fine-grained opinion annotation tasks.

Findings

LLMs achieve high inter-annotator agreement in opinion span identification.

The pipeline reduces variability in manual prompt engineering.

LLMs can serve as effective automatic annotators and adjudicators.

Abstract

Fine-grained opinion analysis of text provides a detailed understanding of expressed sentiments, including the addressed entity. Although this level of detail is sound, it requires considerable human effort and substantial cost to annotate opinions in datasets for training models, especially across diverse domains and real-world applications. We explore the feasibility of LLMs as automatic annotators for fine-grained opinion analysis, addressing the shortage of domain-specific labelled datasets. In this work, we use a declarative annotation pipeline. This approach reduces the variability of manual prompt engineering when using LLMs to identify fine-grained opinion spans in text. We also present a novel methodology for an LLM to adjudicate multiple labels and produce final annotations. After trialling the pipeline with models of different sizes for the Aspect Sentiment Triplet Extraction (ASTE) and Aspect-Category-Opinion-Sentiment (ACOS) analysis tasks, we show that LLMs can serve as automatic annotators and adjudicators, achieving high Inter-Annotator Agreement across individual LLM-based annotators. This reduces the cost and human effort needed to create these fine-grained opinion-annotated datasets.