Contrastive Preference Learning: Learning from Human Feedback without RL

TL;DR

This paper introduces Contrastive Preference Learning (CPL), a novel method for learning from human preferences without reinforcement learning, addressing limitations of existing RLHF approaches in high-dimensional and sequential tasks.

Contribution

The paper proposes CPL, a simple, off-policy algorithm that learns optimal policies directly from preferences using a contrastive objective, avoiding reward modeling and RL.

Findings

CPL scales to high-dimensional, sequential RLHF problems.

CPL outperforms reward-based methods in various settings.

CPL is simpler and more scalable than prior RLHF algorithms.

Abstract

Reinforcement Learning from Human Feedback (RLHF) has emerged as a popular paradigm for aligning models with human intent. Typically RLHF algorithms operate in two phases: first, use human preferences to learn a reward function and second, align the model by optimizing the learned reward via reinforcement learning (RL). This paradigm assumes that human preferences are distributed according to reward, but recent work suggests that they instead follow the regret under the user's optimal policy. Thus, learning a reward function from feedback is not only based on a flawed assumption of human preference, but also leads to unwieldy optimization challenges that stem from policy gradients or bootstrapping in the RL phase. Because of these optimization challenges, contemporary RLHF methods restrict themselves to contextual bandit settings (e.g., as in large language models) or limit observation dimensionality (e.g., state-based robotics). We overcome these limitations by introducing a new family of algorithms for optimizing behavior from human feedback using the regret-based model of human preferences. Using the principle of maximum entropy, we derive Contrastive Preference Learning (CPL), an algorithm for learning optimal policies from preferences without learning reward functions, circumventing the need for RL. CPL is fully off-policy, uses only a simple contrastive objective, and can be applied to arbitrary MDPs. This enables CPL to elegantly scale to high-dimensional and sequential RLHF problems while being simpler than prior methods.