Evaluating the Impact of LLM-Assisted Annotation in a Perspectivized Setting: the Case of FrameNet Annotation

TL;DR

This study evaluates how large language models assist in annotating semantic frames, showing semi-automatic methods improve diversity and coverage, while fully automatic approaches are faster but less comprehensive.

Contribution

It provides an extensive evaluation of LLM-assisted semantic annotation, highlighting the benefits and limitations of semi-automatic versus automatic approaches in a perspectivized NLP setting.

Findings

Semi-automatic annotation increases frame diversity.

Semi-automatic annotation maintains coverage comparable to manual.

Automatic annotation is faster but less accurate.

Abstract

The use of LLM-based applications as a means to accelerate and/or substitute human labor in the creation of language resources and dataset is a reality. Nonetheless, despite the potential of such tools for linguistic research, comprehensive evaluation of their performance and impact on the creation of annotated datasets, especially under a perspectivized approach to NLP, is still missing. This paper contributes to reduction of this gap by reporting on an extensive evaluation of the (semi-)automatization of FrameNet-like semantic annotation by the use of an LLM-based semantic role labeler. The methodology employed compares annotation time, coverage and diversity in three experimental settings: manual, automatic and semi-automatic annotation. Results show that the hybrid, semi-automatic annotation setting leads to increased frame diversity and similar annotation coverage, when compared to the human-only setting, while the automatic setting performs considerably worse in all metrics, except for annotation time.