From Fallback to Frontline: When Can LLMs be Superior Annotators of Human Perspectives?

TL;DR

This paper demonstrates that large language models can often outperform humans in estimating collective human perspectives on subjective tasks, challenging their typical use as mere fallback annotators.

Contribution

It characterizes conditions where LLMs are statistically superior to humans in perspective estimation, highlighting their structural advantages and practical applicability.

Findings

LLMs can outperform humans in predicting subgroup opinions.

Structural properties of LLMs lead to lower variance and bias.

Clear regimes where human judgment remains essential are identified.

Abstract

Although large language models (LLMs) are increasingly used as annotators at scale, they are typically treated as a pragmatic fallback rather than a faithful estimator of human perspectives. This work challenges that presumption. By framing perspective-taking as the estimation of a latent group-level judgment, we characterize the conditions under which modern LLMs can outperform human annotators, including in-group humans, when predicting aggregate subgroup opinions on subjective tasks, and show that these conditions are common in practice. This advantage arises from structural properties of LLMs as estimators, including low variance and reduced coupling between representation and processing biases, rather than any claim of lived experience. Our analysis identifies clear regimes where LLMs act as statistically superior frontline estimators, as well as principled limits where human judgment remains essential. These findings reposition LLMs from a cost-saving compromise to a principled tool for estimating collective human perspectives.