PB$^2$: Preference Space Exploration via Population-Based Methods in Preference-Based Reinforcement Learning

TL;DR

This paper introduces a population-based approach to preference space exploration in preference-based reinforcement learning, improving diversity, robustness, and efficiency in learning from human feedback.

Contribution

It proposes a novel population-based method that enhances exploration and robustness in PbRL, addressing limitations of existing single-agent approaches.

Findings

Enhanced preference exploration with diverse agent populations.

Improved robustness to human feedback errors.

Better performance in complex reward environments.

Abstract

Preference-based reinforcement learning (PbRL) has emerged as a promising approach for learning behaviors from human feedback without predefined reward functions. However, current PbRL methods face a critical challenge in effectively exploring the preference space, often converging prematurely to suboptimal policies that satisfy only a narrow subset of human preferences. In this work, we identify and address this preference exploration problem through population-based methods. We demonstrate that maintaining a diverse population of agents enables more comprehensive exploration of the preference landscape compared to single-agent approaches. Crucially, this diversity improves reward model learning by generating preference queries with clearly distinguishable behaviors, a key factor in real-world scenarios where humans must easily differentiate between options to provide meaningful feedback. Our experiments reveal that current methods may fail by getting stuck in local optima, requiring excessive feedback, or degrading significantly when human evaluators make errors on similar trajectories, a realistic scenario often overlooked by methods relying on perfect oracle teachers. Our population-based approach demonstrates robust performance when teachers mislabel similar trajectory segments and shows significantly enhanced preference exploration capabilities,particularly in environments with complex reward landscapes.