Beyond Majority Voting: Agreement-Based Clustering to Model Annotator Perspectives in Subjective NLP Tasks

TL;DR

This paper introduces an agreement-based clustering method to model annotator disagreement in subjective NLP tasks, improving label aggregation and classification performance across diverse datasets and languages.

Contribution

It presents a novel agreement-based clustering technique that effectively captures annotator perspectives, outperforming traditional majority voting and individual modeling methods.

Findings

Agreement-based clustering improves classification accuracy in subjective NLP tasks.

Multi-label and multitask approaches outperform ensemble and majority vote methods.

The method is effective across 40 datasets in 18 languages and three NLP tasks.

Abstract

Disagreement in annotation is a common phenomenon in the development of NLP datasets and serves as a valuable source of insight. While majority voting remains the dominant strategy for aggregating labels, recent work has explored modeling individual annotators to preserve their perspectives. However, modeling each annotator is resource-intensive and remains underexplored across various NLP tasks. We propose an agreement-based clustering technique to model the disagreement between the annotators. We conduct comprehensive experiments in 40 datasets in 18 typologically diverse languages, covering three subjective NLP tasks: sentiment analysis, emotion classification, and hate speech detection. We evaluate four aggregation approaches: majority vote, ensemble, multi-label, and multitask. The results demonstrate that agreement-based clustering can leverage the full spectrum of annotator perspectives and significantly enhance classification performance in subjective NLP tasks compared to majority voting and individual annotator modeling. Regarding the aggregation approach, the multi-label and multitask approaches are better for modeling clustered annotators than an ensemble and model majority vote.