To Aggregate or Not to Aggregate. That is the Question: A Case Study on Annotation Subjectivity in Span Prediction

TL;DR

This study investigates the impact of annotation subjectivity on span prediction in legal texts, demonstrating that training on consensus annotations yields better performance than using individual annotations.

Contribution

It provides a case study on how annotation subjectivity affects span prediction in legal NLP and shows the benefits of aggregating annotations.

Findings

Training on majority-voted spans improves accuracy.

Subjectivity in annotations influences model performance.

Aggregation of annotations enhances prediction reliability.

Abstract

This paper explores the task of automatic prediction of text spans in a legal problem description that support a legal area label. We use a corpus of problem descriptions written by laypeople in English that is annotated by practising lawyers. Inherent subjectivity exists in our task because legal area categorisation is a complex task, and lawyers often have different views on a problem, especially in the face of legally-imprecise descriptions of issues. Experiments show that training on majority-voted spans outperforms training on disaggregated ones.