Subjective Crowd Disagreements for Subjective Data: Uncovering Meaningful CrowdOpinion with Population-level Learning

TL;DR

This paper introduces CrowdOpinion, an unsupervised learning approach that leverages language features and label distributions to better understand and utilize annotator disagreements in subjective data, improving the analysis of crowd opinions.

Contribution

The paper presents a novel unsupervised clustering method, CrowdOpinion, for analyzing subjective crowd disagreements using language features and label distributions, applicable to various datasets.

Findings

CrowdOpinion effectively pools similar items into larger label distribution samples.

The approach improves understanding of minority and underrepresented opinions.

It performs well across multiple datasets with varying disagreement levels.

Abstract

Human-annotated data plays a critical role in the fairness of AI systems, including those that deal with life-altering decisions or moderating human-created web/social media content. Conventionally, annotator disagreements are resolved before any learning takes place. However, researchers are increasingly identifying annotator disagreement as pervasive and meaningful. They also question the performance of a system when annotators disagree. Particularly when minority views are disregarded, especially among groups that may already be underrepresented in the annotator population. In this paper, we introduce \emph{CrowdOpinion}\footnote{Accepted for publication at ACL 2023}, an unsupervised learning based approach that uses language features and label distributions to pool similar items into larger samples of label distributions. We experiment with four generative and one density-based clustering method, applied to five linear combinations of label distributions and features. We use five publicly available benchmark datasets (with varying levels of annotator disagreements) from social media (Twitter, Gab, and Reddit). We also experiment in the wild using a dataset from Facebook, where annotations come from the platform itself by users reacting to posts. We evaluate \emph{CrowdOpinion} as a label distribution prediction task using KL-divergence and a single-label problem using accuracy measures.