The Energy Loss Phenomenon in RLHF: A New Perspective on Mitigating Reward Hacking

TL;DR

This paper uncovers the energy loss phenomenon in RLHF, linking it to reward hacking, and proposes EPPO, an algorithm that mitigates reward hacking by penalizing energy loss, supported by theoretical and empirical evidence.

Contribution

It introduces the energy loss phenomenon in RLHF, provides a theoretical foundation, and proposes EPPO, a novel method to mitigate reward hacking by controlling energy loss.

Findings

Energy loss increases during RLHF and correlates with reward hacking.

EPPO effectively reduces reward hacking across various LLMs.

Theoretical analysis links energy loss control to entropy-regularized RL.

Abstract

This work identifies the Energy Loss Phenomenon in Reinforcement Learning from Human Feedback (RLHF) and its connection to reward hacking. Specifically, energy loss in the final layer of a Large Language Model (LLM) gradually increases during the RL process, with an excessive increase in energy loss characterizing reward hacking. Beyond empirical analysis, we further provide a theoretical foundation by proving that, under mild conditions, the increased energy loss reduces the upper bound of contextual relevance in LLMs, which is a critical aspect of reward hacking as the reduced contextual relevance typically indicates overfitting to reward model-favored patterns in RL. To address this issue, we propose an Energy loss-aware PPO algorithm (EPPO) which penalizes the increase in energy loss in the LLM's final layer during reward calculation to prevent excessive energy loss, thereby mitigating reward hacking. We theoretically show that EPPO can be conceptually interpreted as an entropy-regularized RL algorithm, which provides deeper insights into its effectiveness. Extensive experiments across various LLMs and tasks demonstrate the commonality of the energy loss phenomenon, as well as the effectiveness of EPPO in mitigating reward hacking and improving RLHF performance.