Learning a Pessimistic Reward Model in RLHF

TL;DR

This paper introduces PET, a new method for fine-tuning reward models in RLHF that prevents reward hacking by adopting a pessimistic approach, enabling high-quality policy learning without regularization.

Contribution

The paper presents PET, a novel pessimistic reward fine-tuning technique that effectively mitigates reward hacking without regularization in offline RLHF settings.

Findings

High-quality policies can be learned without regularization.

Pessimistic reward models prevent reward hacking.

Policies with high KL divergence can still perform well.

Abstract

This work proposes `PET', a novel pessimistic reward fine-tuning method, to learn a pessimistic reward model robust against reward hacking in offline reinforcement learning from human feedback (RLHF). Traditional reward modeling techniques in RLHF train an imperfect reward model, on which a KL regularization plays a pivotal role in mitigating reward hacking when optimizing a policy. Such an intuition-based method still suffers from reward hacking, and the policies with large KL divergence from the dataset distribution are excluded during learning. In contrast, we show that when optimizing a policy on a pessimistic reward model fine-tuned through PET, reward hacking can be prevented without relying on any regularization. We test our methods on the standard TL;DR summarization dataset. We find that one can learn a high-quality policy on our pessimistic reward without using any regularization. Such a policy has a high KL divergence from the dataset distribution while having high performance in practice. In summary, our work shows the feasibility of learning a pessimistic reward model against reward hacking. The agent can greedily search for the policy with a high pessimistic reward without suffering from reward hacking.