Perspectives in Play: A Multi-Perspective Approach for More Inclusive NLP Systems

TL;DR

This paper introduces a multi-perspective approach using soft labels in NLP to better capture human disagreement and improve inclusivity in subjective text classification tasks, outperforming traditional aggregation methods.

Contribution

It proposes a novel multi-perspective method with soft labels for more inclusive NLP models, addressing the underrepresentation of minority viewpoints in subjective tasks.

Findings

Better approximation of human label distributions (lower JSD)

Higher F1 scores compared to traditional methods

Lower confidence in highly subjective tasks like irony detection

Abstract

In the realm of Natural Language Processing (NLP), common approaches for handling human disagreement consist of aggregating annotators' viewpoints to establish a single ground truth. However, prior studies show that disregarding individual opinions can lead can lead to the side effect of underrepresenting minority perspectives, especially in subjective tasks, where annotators may systematically disagree because of their preferences. Recognizing that labels reflect the diverse backgrounds, life experiences, and values of individuals, this study proposes a new multi-perspective approach using soft labels to encourage the development of the next generation of perspective aware models, more inclusive and pluralistic. We conduct an extensive analysis across diverse subjective text classification tasks, including hate speech, irony, abusive language, and stance detection, to highlight the importance of capturing human disagreements, often overlooked by traditional aggregation methods. Results show that the multi-perspective approach not only better approximates human label distributions, as measured by Jensen-Shannon Divergence (JSD), but also achieves superior classification performance (higher F1 scores), outperforming traditional approaches. However, our approach exhibits lower confidence in tasks like irony and stance detection, likely due to the inherent subjectivity present in the texts. Lastly, leveraging Explainable AI (XAI), we explore model uncertainty and uncover meaningful insights into model predictions.