Aggregation Artifacts in Subjective Tasks Collapse Large Language Models' Posteriors

TL;DR

This paper investigates how aggregation artifacts in datasets affect Large Language Models' performance on subjective tasks, revealing that modeling individual annotators can improve alignment and understanding of model biases.

Contribution

The study demonstrates that dataset aggregation introduces noise affecting LLMs' handling of subjective tasks and advocates for modeling individual annotator perspectives instead.

Findings

Aggregation artifacts introduce noise in subjective task datasets.

Modeling individual annotators improves LLM alignment with diverse perspectives.

Aggregation does not fully account for the gap between ICL and state-of-the-art performance.

Abstract

In-context Learning (ICL) has become the primary method for performing natural language tasks with Large Language Models (LLMs). The knowledge acquired during pre-training is crucial for this few-shot capability, providing the model with task priors. However, recent studies have shown that ICL predominantly relies on retrieving task priors rather than "learning" to perform tasks. This limitation is particularly evident in complex subjective domains such as emotion and morality, where priors significantly influence posterior predictions. In this work, we examine whether this is the result of the aggregation used in corresponding datasets, where trying to combine low-agreement, disparate annotations might lead to annotation artifacts that create detrimental noise in the prompt. Moreover, we evaluate the posterior bias towards certain annotators by grounding our study in appropriate, quantitative measures of LLM priors. Our results indicate that aggregation is a confounding factor in the modeling of subjective tasks, and advocate focusing on modeling individuals instead. However, aggregation does not explain the entire gap between ICL and the state of the art, meaning other factors in such tasks also account for the observed phenomena. Finally, by rigorously studying annotator-level labels, we find that it is possible for minority annotators to both better align with LLMs and have their perspectives further amplified.