Bridging the Gap: In-Context Learning for Modeling Human Disagreement

TL;DR

This paper investigates how large language models can better reflect human disagreement in subjective NLP tasks through in-context learning, emphasizing the importance of prompt design and demonstration selection.

Contribution

It demonstrates the potential and limitations of using in-context learning to model multiple human perspectives and disagreements in subjective tasks.

Findings

Multi-perspective generation is feasible in zero-shot settings.

Few-shot setups often do not capture the full range of human judgments.

Prompt design and demonstration selection significantly influence performance.

Abstract

Large Language Models (LLMs) have shown strong performance on NLP classification tasks. However, they typically rely on aggregated labels-often via majority voting-which can obscure the human disagreement inherent in subjective annotations. This study examines whether LLMs can capture multiple perspectives and reflect annotator disagreement in subjective tasks such as hate speech and offensive language detection. We use in-context learning (ICL) in zero-shot and few-shot settings, evaluating four open-source LLMs across three label modeling strategies: aggregated hard labels, and disaggregated hard and soft labels. In few-shot prompting, we assess demonstration selection methods based on textual similarity (BM25, PLM-based), annotation disagreement (entropy), a combined ranking, and example ordering strategies (random vs. curriculum-based). Results show that multi-perspective generation is viable in zero-shot settings, while few-shot setups often fail to capture the full spectrum of human judgments. Prompt design and demonstration selection notably affect performance, though example ordering has limited impact. These findings highlight the challenges of modeling subjectivity with LLMs and the importance of building more perspective-aware, socially intelligent models.