Optimal Design for Reward Modeling in RLHF

TL;DR

This paper formalizes the reward training process in RLHF, framing dataset selection as a simple regret minimization problem, and provides theoretical bounds and guarantees for optimal reward modeling.

Contribution

It introduces a novel offline framework for reward model training in RLHF with theoretical guarantees, addressing the cost of collecting human preferences.

Findings

Derived bounds on simple regret for reward models

Proposed an offline approach for dataset selection in RLHF

Established a lower bound matching the upper bound

Abstract

Reinforcement Learning from Human Feedback (RLHF) has become a popular approach to align language models (LMs) with human preferences. This method involves collecting a large dataset of human pairwise preferences across various text generations and using it to infer (implicitly or explicitly) a reward model. Numerous methods have been proposed to learn the reward model and align a LM with it. However, the costly process of collecting human preferences has received little attention and could benefit from theoretical insights. This paper addresses this issue and aims to formalize the reward training model in RLHF. We frame the selection of an effective dataset as a simple regret minimization task, using a linear contextual dueling bandit method. Given the potentially large number of arms, this approach is more coherent than the best-arm identification setting. We then propose an offline framework for solving this problem. Under appropriate assumptions - linearity of the reward model in the embedding space, and boundedness of the reward parameter - we derive bounds on the simple regret. Finally, we provide a lower bound that matches our upper bound up to constant and logarithmic terms. To our knowledge, this is the first theoretical contribution in this area to provide an offline approach as well as worst-case guarantees.