Influencing Humans to Conform to Preference Models for RLHF

TL;DR

This paper explores methods to influence how humans express preferences to better align with RLHF models, improving the quality of preference data without altering the underlying reward function.

Contribution

It introduces three interventions—visualizing underlying quantities, training for specific models, and modifying questions—to enhance human conformance to preference models in RLHF.

Findings

All interventions significantly affected human preference expression.

Interventions improved the alignment of human preferences with assumed models.

Practical tools were developed to enhance preference data quality for RLHF.

Abstract

Designing a reinforcement learning from human feedback (RLHF) algorithm to approximate a human's unobservable reward function requires assuming, implicitly or explicitly, a model of human preferences. A preference model that poorly describes how humans generate preferences risks learning a poor approximation of the human's reward function. In this paper, we conduct three human studies to asses whether one can influence the expression of real human preferences to more closely conform to a desired preference model. Importantly, our approach does not seek to alter the human's unobserved reward function. Rather, we change how humans use this reward function to generate preferences, such that they better match whatever preference model is assumed by a particular RLHF algorithm. We introduce three interventions: showing humans the quantities that underlie a preference model, which is normally unobservable information derived from the reward function; training people to follow a specific preference model; and modifying the preference elicitation question. All intervention types show significant effects, providing practical tools to improve preference data quality and the resultant alignment of the learned reward functions. Overall we establish a novel research direction in model alignment: designing interfaces and training interventions to increase human conformance with the modeling assumptions of the algorithm that will learn from their input.