The Expertise Problem: Learning from Specialized Feedback

TL;DR

This paper formalizes the expertise problem in reinforcement learning from human feedback, highlighting how varying teacher expertise affects feedback reliability, and provides an open-source benchmark for future research.

Contribution

It introduces the expertise problem in RLHF, extends an existing benchmark to evaluate it, and offers techniques for improved teacher and query selection.

Findings

Demonstrates the impact of teacher expertise variability on RLHF performance.

Provides an open-source benchmark for the expertise problem.

Evaluates state-of-the-art RLHF algorithms under expertise variability.

Abstract

Reinforcement learning from human feedback (RLHF) is a powerful technique for training agents to perform difficult-to-specify tasks. However, human feedback can be noisy, particularly when human teachers lack relevant knowledge or experience. Levels of expertise vary across teachers, and a given teacher may have differing levels of expertise for different components of a task. RLHF algorithms that learn from multiple teachers therefore face an expertise problem: the reliability of a given piece of feedback depends both on the teacher that it comes from and how specialized that teacher is on relevant components of the task. Existing state-of-the-art RLHF algorithms assume that all evaluations come from the same distribution, obscuring this inter- and intra-human variance, and preventing them from accounting for or taking advantage of variations in expertise. We formalize this problem, implement it as an extension of an existing RLHF benchmark, evaluate the performance of a state-of-the-art RLHF algorithm, and explore techniques to improve query and teacher selection. Our key contribution is to demonstrate and characterize the expertise problem, and to provide an open-source implementation for testing future solutions.