How Humans Help LLMs: Assessing and Incentivizing Human Preference Annotators

TL;DR

This paper investigates methods to monitor and incentivize human annotators for preference data in LLM training, proposing a self-consistency scheme and analyzing sample complexity for effective quality control.

Contribution

It introduces a self-consistency monitoring method tailored for preference annotation and analyzes the sample complexity of incentivization contracts.

Findings

Self-consistency monitoring can outperform expert-based monitoring under certain conditions.

Linear contracts are rate-optimal among general contracts for incentivizing annotators.

The sample complexity for effective incentivization scales as 1/(I n) for linear contracts.

Abstract

Human-annotated preference data play an important role in aligning large language models (LLMs). In this paper, we study two connected questions: how to monitor the quality of human preference annotators and how to incentivize them to provide high-quality annotations. In current practice, expert-based monitoring is a natural workhorse for quality control, but it performs poorly in preference annotation because annotators are heterogeneous and downstream model performance is an indirect and noisy proxy for annotation quality. We therefore propose a self-consistency monitoring scheme tailored to preference annotation, and analyze the statistical sample complexity of both methods. This practitioner-facing analysis identifies how many inspected samples are needed to reliably assess an annotator and shows when self-consistency monitoring can outperform expert-based monitoring. We then use the resulting monitoring signal as the performance measure in a principal-agent model, which lets us study a second sample-complexity question: how many monitored samples are needed before simple contracts perform close to the ideal benchmark in which annotation quality is perfectly observable. Under this continuous action space, we show that this shortfall scales as $\Theta(1/\sqrt{\mathcal{I} n \log n})$ for binary contracts and $\Theta(1/(\mathcal{I}n))$ for linear contracts, where $\mathcal{I}$ is the Fisher information and $n$ is the number of samples; we further show that the linear contracts are rate-optimal among general contracts. This contrasts with the known result that binary contracts are optimal and of $\exp(-\Theta(n))$ when the action space is discrete \citep{frick2023monitoring}.