Exploring and Addressing Reward Confusion in Offline Preference Learning

TL;DR

This paper investigates reward confusion in offline RLHF caused by spurious correlations, introduces a benchmark for this problem, and proposes a method leveraging preference transitivity to mitigate confusion.

Contribution

It presents a new benchmark for reward confusion in offline RLHF and a method using preference transitivity and active learning to reduce reward confusion.

Findings

The proposed method significantly reduces reward confusion.

Benchmark effectively evaluates reward confusion issues.

Preference transitivity improves reward model reliability.

Abstract

Spurious correlations in a reward model's training data can prevent Reinforcement Learning from Human Feedback (RLHF) from identifying the desired goal and induce unwanted behaviors. This paper shows that offline RLHF is susceptible to reward confusion, especially in the presence of spurious correlations in offline data. We create a benchmark to study this problem and propose a method that can significantly reduce reward confusion by leveraging transitivity of preferences while building a global preference chain with active learning.