Utility-inspired Reward Transformations Improve Reinforcement Learning Training of Language Models

TL;DR

This paper introduces a utility-inspired reward transformation for reinforcement learning in language models, improving training by better handling reward sensitivities and dependencies, leading to more helpful and less harmful outputs.

Contribution

It proposes a novel Inada-inspired reward transformation that addresses limitations of linear reward aggregation in RL training of language models.

Findings

Inada-inspired rewards improve helpfulness of generated text

Models trained with this method are less harmful

Outperforms traditional reward averaging in experiments

Abstract

Current methods that train large language models (LLMs) with reinforcement learning feedback, often resort to averaging outputs of multiple rewards functions during training. This overlooks crucial aspects of individual reward dimensions and inter-reward dependencies that can lead to sub-optimal outcomes in generations. In this work, we show how linear aggregation of rewards exhibits some vulnerabilities that can lead to undesired properties of generated text. We then propose a transformation of reward functions inspired by economic theory of utility functions (specifically Inada conditions), that enhances sensitivity to low reward values while diminishing sensitivity to already high values. We compare our approach to the existing baseline methods that linearly aggregate rewards and show how the Inada-inspired reward feedback is superior to traditional weighted averaging. We quantitatively and qualitatively analyse the difference in the methods, and see that models trained with Inada-transformations score as more helpful while being less harmful.