SemEval-2023 Task 11: Learning With Disagreements (LeWiDi)

TL;DR

The LeWiDi shared task promotes preserving annotator disagreements in NLP datasets, especially for subjective tasks, by providing a framework for training and evaluating models that embrace such disagreements.

Contribution

This paper introduces the second LeWiDi shared task focusing on NLP, emphasizing subjective tasks and soft evaluation methods to better handle annotator disagreements.

Findings

Attracted 13 submissions from diverse participants.

Focused exclusively on NLP and subjective tasks.

Highlighted the importance of soft evaluation approaches.

Abstract

NLP datasets annotated with human judgments are rife with disagreements between the judges. This is especially true for tasks depending on subjective judgments such as sentiment analysis or offensive language detection. Particularly in these latter cases, the NLP community has come to realize that the approach of 'reconciling' these different subjective interpretations is inappropriate. Many NLP researchers have therefore concluded that rather than eliminating disagreements from annotated corpora, we should preserve them-indeed, some argue that corpora should aim to preserve all annotator judgments. But this approach to corpus creation for NLP has not yet been widely accepted. The objective of the LeWiDi series of shared tasks is to promote this approach to developing NLP models by providing a unified framework for training and evaluating with such datasets. We report on the second LeWiDi shared task, which differs from the first edition in three crucial respects: (i) it focuses entirely on NLP, instead of both NLP and computer vision tasks in its first edition; (ii) it focuses on subjective tasks, instead of covering different types of disagreements-as training with aggregated labels for subjective NLP tasks is a particularly obvious misrepresentation of the data; and (iii) for the evaluation, we concentrate on soft approaches to evaluation. This second edition of LeWiDi attracted a wide array of participants resulting in 13 shared task submission papers.