Annotator-Centric Active Learning for Subjective NLP Tasks

TL;DR

This paper proposes Annotator-Centric Active Learning (ACAL), a method that improves data efficiency and captures diverse human judgments in subjective NLP tasks by strategically selecting annotators.

Contribution

It introduces a novel annotator selection strategy within active learning tailored for subjective NLP tasks, emphasizing diversity and annotator-centric evaluation.

Findings

ACAL enhances data efficiency in subjective NLP tasks.

ACAL performs well in annotator-centric evaluation metrics.

Success depends on having a large, diverse pool of annotators.

Abstract

Active Learning (AL) addresses the high costs of collecting human annotations by strategically annotating the most informative samples. However, for subjective NLP tasks, incorporating a wide range of perspectives in the annotation process is crucial to capture the variability in human judgments. We introduce Annotator-Centric Active Learning (ACAL), which incorporates an annotator selection strategy following data sampling. Our objective is two-fold: 1) to efficiently approximate the full diversity of human judgments, and 2) to assess model performance using annotator-centric metrics, which value minority and majority perspectives equally. We experiment with multiple annotator selection strategies across seven subjective NLP tasks, employing both traditional and novel, human-centered evaluation metrics. Our findings indicate that ACAL improves data efficiency and excels in annotator-centric performance evaluations. However, its success depends on the availability of a sufficiently large and diverse pool of annotators to sample from.