Adaptive Preference Scaling for Reinforcement Learning with Human Feedback

TL;DR

This paper introduces an adaptive preference scaling method for reinforcement learning with human feedback, improving reward modeling by accounting for varying preference strengths, leading to better policy performance and easier hyperparameter tuning.

Contribution

It proposes a novel adaptive preference loss based on distributionally robust optimization that dynamically adjusts to preference ambiguity, enhancing reward function flexibility.

Findings

Improves policy performance in robotic control and language generation tasks.

Aligns reward functions more closely with policy optimization.

Simplifies hyperparameter tuning process.

Abstract

Reinforcement learning from human feedback (RLHF) is a prevalent approach to align AI systems with human values by learning rewards from human preference data. Due to various reasons, however, such data typically takes the form of rankings over pairs of trajectory segments, which fails to capture the varying strengths of preferences across different pairs. In this paper, we propose a novel adaptive preference loss, underpinned by distributionally robust optimization (DRO), designed to address this uncertainty in preference strength. By incorporating an adaptive scaling parameter into the loss for each pair, our method increases the flexibility of the reward function. Specifically, it assigns small scaling parameters to pairs with ambiguous preferences, leading to more comparable rewards, and large scaling parameters to those with clear preferences for more distinct rewards. Computationally, our proposed loss function is strictly convex and univariate with respect to each scaling parameter, enabling its efficient optimization through a simple second-order algorithm. Our method is versatile and can be readily adapted to various preference optimization frameworks, including direct preference optimization (DPO). Our experiments with robotic control and natural language generation with large language models (LLMs) show that our method not only improves policy performance but also aligns reward function selection more closely with policy optimization, simplifying the hyperparameter tuning process.