Multi-Perspective Stance Detection

TL;DR

This study explores how incorporating multiple annotator perspectives in stance detection improves model accuracy and confidence, emphasizing the importance of inclusive, perspective-aware AI models for ethical and effective NLP applications.

Contribution

It introduces a perspective-aware classification approach that leverages multiple annotations, demonstrating improved performance over traditional single-label methods.

Findings

Multi-perspective approach outperforms baseline models.

Including diverse annotations enhances classification accuracy.

Perspective-aware models increase model confidence.

Abstract

Subjective NLP tasks usually rely on human annotations provided by multiple annotators, whose judgments may vary due to their diverse backgrounds and life experiences. Traditional methods often aggregate multiple annotations into a single ground truth, disregarding the diversity in perspectives that arises from annotator disagreement. In this preliminary study, we examine the effect of including multiple annotations on model accuracy in classification. Our methodology investigates the performance of perspective-aware classification models in stance detection task and further inspects if annotator disagreement affects the model confidence. The results show that multi-perspective approach yields better classification performance outperforming the baseline which uses the single label. This entails that designing more inclusive perspective-aware AI models is not only an essential first step in implementing responsible and ethical AI, but it can also achieve superior results than using the traditional approaches.