The Accuracy Paradox in RLHF: When Better Reward Models Don't Yield Better Language Models

TL;DR

This paper reveals that in RLHF for language models, using moderately accurate reward models can outperform highly accurate ones, challenging the assumption that better reward models always produce better language models.

Contribution

It uncovers the paradox that stronger reward models do not always lead to improved language model performance, based on experiments with the QA-FEEDBACK dataset.

Findings

Moderately accurate reward models outperform highly accurate ones in training.

The paradox challenges existing beliefs about reward model strength and model quality.

Results suggest the need to reconsider reward model selection strategies.

Abstract

Reinforcement Learning from Human Feedback significantly enhances Natural Language Processing by aligning language models with human expectations. A critical factor in this alignment is the strength of reward models used during training. This study explores whether stronger reward models invariably lead to better language models. In this paper, through experiments on relevance, factuality, and completeness tasks using the QA-FEEDBACK dataset and reward models based on Longformer, we uncover a surprising paradox: language models trained with moderately accurate reward models outperform those guided by highly accurate ones. This challenges the widely held belief that stronger reward models always lead to better language models, and opens up new avenues for future research into the key factors driving model performance and how to choose the most suitable reward models. Code and additional details are available at https://github.com/EIT-NLP/AccuracyParadox-RLHF.