Bias Fitting to Mitigate Length Bias of Reward Model in RLHF

TL;DR

This paper introduces FiMi-RM, a framework that learns and corrects length bias in reward models used in RLHF, leading to more balanced responses and improved alignment without verbosity.

Contribution

FiMi-RM is the first method to explicitly model and mitigate non-linear length bias in reward models for RLHF, enhancing alignment quality.

Findings

Reduces verbosity in language model responses.

Improves length-controlled win rate.

Balances length-reward distribution.

Abstract

Reinforcement Learning from Human Feedback relies on reward models to align large language models with human preferences. However, RLHF often suffers from reward hacking, wherein policy learning exploits flaws in the trained reward model to maximize reward scores without genuinely aligning with human preferences. A significant example of such reward hacking is length bias, where reward models usually favor longer responses irrespective of actual response quality. Previous works on length bias have notable limitations, these approaches either mitigate bias without characterizing the bias form, or simply assume a linear length-reward relation. To accurately model the intricate nature of length bias and facilitate more effective bias mitigation, we propose FiMi-RM (Bias Fitting to Mitigate Length Bias of Reward Model in RLHF), a framework that autonomously learns and corrects underlying bias patterns. Our approach consists of three stages: First, we train a standard reward model which inherently contains length bias. Next, we deploy a lightweight fitting model to explicitly capture the non-linear relation between length and reward. Finally, we incorporate this learned relation into the reward model to debias. Experimental results demonstrate that FiMi-RM achieves a more balanced length-reward distribution. Furthermore, when applied to alignment algorithms, our debiased reward model improves length-controlled win rate and reduces verbosity without compromising its performance.