Multi-Perspective LLM Annotations for Valid Analyses in Subjective Tasks

TL;DR

This paper introduces Perspective-Driven Inference, a method for improving LLM annotations in subjective tasks by focusing on demographic group disagreements, using minimal human annotation to enhance accuracy.

Contribution

It presents a novel approach that models annotation distributions across groups and employs adaptive sampling to target difficult demographic groups in subjective tasks.

Findings

Improved annotation accuracy for challenging demographic groups.

Maintained coverage while reducing human annotation effort.

Effective in politeness and offensiveness rating tasks.

Abstract

Large language models are increasingly used to annotate texts, but their outputs reflect some human perspectives better than others. Existing methods for correcting LLM annotation error assume a single ground truth. However, this assumption fails in subjective tasks where disagreement across demographic groups is meaningful. Here we introduce Perspective-Driven Inference, a method that treats the distribution of annotations across groups as the quantity of interest, and estimates it using a small human annotation budget. We contribute an adaptive sampling strategy that concentrates human annotation effort on groups where LLM proxies are least accurate. We evaluate on politeness and offensiveness rating tasks, showing targeted improvements for harder-to-model demographic groups relative to uniform sampling baselines, while maintaining coverage.