Provable Benefits of Policy Learning from Human Preferences in Contextual Bandit Problems

TL;DR

This paper provides a theoretical analysis of human feedback methods in contextual bandit problems, comparing absolute ratings and preference-based feedback, and explains their empirical success through bias and uncertainty considerations.

Contribution

It offers a theoretical comparison of human feedback approaches in offline contextual bandits, highlighting how bias and uncertainty influence their guarantees.

Findings

Preference-based methods have favorable theoretical guarantees.

Bias and uncertainty in human feedback impact reward learning.

Preference methods explain empirical successes in practice.

Abstract

For a real-world decision-making problem, the reward function often needs to be engineered or learned. A popular approach is to utilize human feedback to learn a reward function for training. The most straightforward way to do so is to ask humans to provide ratings for state-action pairs on an absolute scale and take these ratings as reward samples directly. Another popular way is to ask humans to rank a small set of state-action pairs by preference and learn a reward function from these preference data. Recently, preference-based methods have demonstrated substantial success in empirical applications such as InstructGPT. In this work, we develop a theoretical comparison between these human feedback approaches in offline contextual bandits and show how human bias and uncertainty in feedback modelings can affect the theoretical guarantees of these approaches. Through this, our results seek to provide a theoretical explanation for the empirical successes of preference-based methods from a modeling perspective.