Three Models of RLHF Annotation: Extension, Evidence, and Authority

TL;DR

This paper clarifies three conceptual models of human judgment roles in RLHF, analyzing their implications and proposing tailored annotation pipelines for improved alignment of language models.

Contribution

It introduces and distinguishes three models of human judgment in RLHF—extension, evidence, and authority—and offers normative criteria for their application.

Findings

Survey of landmark RLHF papers illustrating implicit use of models

Identification of failure modes from conflating models

Recommendation to decompose annotation into separable dimensions

Abstract

Preference-based alignment methods, most prominently Reinforcement Learning with Human Feedback (RLHF), use the judgments of human annotators to shape large language model behaviour. However, the normative role of these judgments is rarely made explicit. I distinguish three conceptual models of that role. The first is extension: annotators extend the system designers' own judgments about what outputs should be. The second is evidence: annotators provide independent evidence about some facts, whether moral, social or otherwise. The third is authority: annotators have some independent authority (as representatives of the broader population) to determine system outputs. I argue that these models have implications for how RLHF pipelines should solicit, validate and aggregate annotations. I survey landmark papers in the literature on RLHF and related methods to illustrate how they implicitly draw on these models, describe failure modes that come from unintentionally or intentionally conflating them, and offer normative criteria for choosing among them. My central recommendation is that RLHF pipeline designers should decompose annotation into separable dimensions and tailor each pipeline to the model most appropriate for that dimension, rather than seeking a single unified pipeline.