RED: Unleashing Token-Level Rewards from Holistic Feedback via Reward Redistribution

TL;DR

RED introduces a token-level reward redistribution method for reinforcement learning from human feedback, providing more precise guidance without additional training costs, leading to improved language model alignment.

Contribution

The paper presents a novel token-level reward redistribution technique that enhances RLHF by assigning specific credit to individual tokens without modifying the reward model.

Findings

Outperforms existing methods across multiple datasets and tasks.

Requires no additional training or modifications to the reward model.

Improves language model alignment with human preferences.

Abstract

Reinforcement learning from human feedback (RLHF) offers a promising approach to aligning large language models (LLMs) with human preferences. Typically, a reward model is trained or supplied to act as a proxy for humans in evaluating generated responses during the reinforcement training phase. However, current reward models operate as sequence-to-one models, allocating a single, sparse, and delayed reward to an entire output sequence. This approach may overlook the significant contributions of individual tokens toward the desired outcome. To this end, we propose a more fine-grained, token-level guidance approach for RL training. Specifically, we introduce RED, a novel reward redistribition method that evaluates and assigns specific credit to each token using an off-the-shelf reward model. Utilizing these fine-grained rewards enhances the model's understanding of language nuances, leading to more precise performance improvements. Notably, our method does not require modifying the reward model or introducing additional training steps, thereby incurring minimal computational costs. Experimental results across diverse datasets and tasks demonstrate the superiority of our approach.